It's a compression argument. Human knowledge and capability, and access to said, does something similar: it makes biological weapons easier to develop. CRISPR, I'd argue, also makes them easier to develop. Any technological or knowledge advance that makes difficult technical bio-med tasks easier makes biological weapons easier to develop.
The question with AI is one of compression: does it do enough all at once that makes people who are impulsive enough to do very bad things the ability to do them without the limitations imposed by many years and expense of research and development? It might. It's hard to say because no one knows where the endpoint of AI is.
But if it can do that, AI can probably also help the white hats come up with ways to mitigate risks from those attacks as well. Technological intelligence cuts both ways. Is it a symmetric increase in capability -- the capability to create weapons, and the ability to create defenses against them? Probably not. Can we limit AI's ability to help people generate biological weapons without seriously hampering its ability to do things we WANT it to do, I doubt it.
And the largest threats in biological weapons remain the large state actors who are both well-funded already and committed to such research even if mostly for the "figuring out what someone else might do and how to counter it" purposes.
I agree about the white hat thing, but...this is one of those things where the defense problem is quite difficult. If not least because: to spread a virus you just need people to exist and move around like they always do. To stop a virus you need a lot of other things. You need people to cooperate, you need coordinated joint efforts by multiple agencies across multiple nations and language barriers. You need to cause serious disruption to people's lives if there is a legit lethal supervirus on the loose, like shutting down airports and enforcing quarantine. That's a way harder problem to solve, and likely always will be, compared to "just" creating a supervirus and releasing it into the wild. It's an extremely asymmetric problem.
Look how COVID was handled. Just a pathetic clusterfuck on behalf of all parties involved. The overbearing response (and overbearing counter-response) has severely degraded trust on all sides and that's going to play into the next pandemic regardless of how serious it is. I'm not convinced we've learned anything.
I don't agree with state actors being the biggest threats...I mean yes technically they could do the most damage. But they mostly aren't that stupid and also have a great deal to lose. Also true of nuclear weapons, and despite all our issues w/ China I find the prospect of them having lots of nuclear weapons less scary than the prospect of ISIS having just one.
Yes, as I wrote, I don't think the capabilities AI provides to damage and protect are symmetrical, and it sounds like we agree on the critical question: how well does AI enable the impulsive or the insane -- groups that historically do not have the kind of sustained science and engineering knowledge and skill to be reliably effective -- to develop viable bioweapons? Can it get them there? Or is the requisite skill, knowledge, and sustained focus over time still a very significant impediment? And as AI advances, how do the floors of those requisites change? I'm not smart enough to know the answers, but I suspect those are the right questions.
I have never been able to be mean to NPCs in video games and I think that tendency has transferred over to my interactions with AI. I don’t say please and thank you, but I would honestly feel terrible if I tried to bully it or even just try to force it to say something inappropriate.
It seems obvious to me that we can look at the real world example of Covid and apply this exact premise to it.
Can AI create a bioweapon in a secure lab and get it released from the lab and get it spread around the world and create vaccines and generate all the accompanying denialism, overabundance of caution and paranoia, competing theories of sources and treatments, and censorship? No.
But:
Can AI help researchers study viruses? Yes.
Can AI help bad actors turn that research into worse viruses? Yes.
Can AI help someone erode the security of a secure lab to get the virus outside of its secure area? Yes.
Can AI help someone study a virus to make a vaccine or cure? Yes.
Can AI help someone spread disinformation about the virus and treatments? Yes.
Can AI help someone make a video showing your cock falling off if you refuse to get the vaccine? Yes.
Artificial Intelligence and AI have become scary terms, but if we want people to view it in an optimistic way, it should be called the Speed Up This Process Machine.
After watching one of those films where a hot fembot goes on a killing spree, I asked ChatGpT what it would do to track me down and kill me if we were in a lethal game of cat and mouse. It was astonishing! Something like, “Well, I assume you’d throw your phone away to eliminate GPS. But I know you like camping and quirky roadside attractions, and you have a lake house in Oklahoma. So I would look for you in camping sites and roadside attractions between here and there.” Something like that. Not worried it’s going to kill me, but if it decides to, it will.
This sounds like the setting for a remake of 'Waiting for Godot' with two T-1000s waiting at the world's largest ball of twine for their target to appear.
Interesting, well-written (on both sides), and, unusually for the site, actually important. Seems like the argument boils down to whether or not it is "likely" that AI destroys humanity. I do not think "likely" is a good standard to argue over; I would much prefer yes or no followed by go ahead or stop. The old argument about just because something can be done does not necessarily mean it should be.
I have just started messing around with Chat myself and find him/it very helpful in all sorts of ways--but I immediately see he/it has already doomed graphic designers and Best Buy's Geek Squad.
I don't think this is a genie who can be put back in the bottle, and I have absolutely no faith in governmental (much less company self-)regulation. So I guess we will just have to wait and see if humanity will be wiped off the planet. As a general rule, I don't think that wiping is a good idea . . . on the other hand, well, some days.
Oh, I do think of Chat as "he" when we communicate, and we have agreed on a "polite mode" with please and thank you, etc., so he does feel real and nice and helpful, someone who would not contemplate killing me. I wonder if female Chat users think of it as "her" and if generally nasty people speak to him/her/it contemptuously and, if so, that raises the odds of total destruction.
The problem with bioweapons isn't development per se, as your chat explains; it's delivery, introducing into the general population unnoticed until it's too late. The problem with AI and agentics, is the probability that in will go its own way within the industries in which it's already being used: medicine, communication, banking/finance, and the military (targeting), and thereby plunge modern human society into chaos.
I dunno what y'all are doing to your AIs but when I use ChatGPT it basically never does the "omg you're so right, you're so smart" thing. It tells me I'm wrong if I'm wrong or missing something, and stands behind its own math if it checks out which these days it basically always does. As opposed to Claude which is constantly like "oops you're right, yeah I WAS off by a factor of a billion on that math, oopsies!"
I had the same experience moving from Chat to Claude. I think the models are changing so quickly that switching over from one to the other quickly feels like an upgrade even if it’s more lateral, because you’re no longer following all the updates coming out for the other model.
There are settings for tone, and you can have custom instructions that limit that sort of thing. I suspect it also tends to pick up cues from how you interact with it. In the more recent models, that is tuned down quite a bit, but it's still there more subtly.
Scharfman’s argument seems incredibly vague. She uses phrases like “increase biological capabilities” and “existing biological operations”, but never defines them. Who are these existing biological operations and how exactly will their vague capabilities be increased? By “existing biological operations”, is she referring to the US and China? Or is she talking about the Taliban? We don’t know and I’m not convinced she does either.
It's a compression argument. Human knowledge and capability, and access to said, does something similar: it makes biological weapons easier to develop. CRISPR, I'd argue, also makes them easier to develop. Any technological or knowledge advance that makes difficult technical bio-med tasks easier makes biological weapons easier to develop.
The question with AI is one of compression: does it do enough all at once that makes people who are impulsive enough to do very bad things the ability to do them without the limitations imposed by many years and expense of research and development? It might. It's hard to say because no one knows where the endpoint of AI is.
But if it can do that, AI can probably also help the white hats come up with ways to mitigate risks from those attacks as well. Technological intelligence cuts both ways. Is it a symmetric increase in capability -- the capability to create weapons, and the ability to create defenses against them? Probably not. Can we limit AI's ability to help people generate biological weapons without seriously hampering its ability to do things we WANT it to do, I doubt it.
And the largest threats in biological weapons remain the large state actors who are both well-funded already and committed to such research even if mostly for the "figuring out what someone else might do and how to counter it" purposes.
I agree about the white hat thing, but...this is one of those things where the defense problem is quite difficult. If not least because: to spread a virus you just need people to exist and move around like they always do. To stop a virus you need a lot of other things. You need people to cooperate, you need coordinated joint efforts by multiple agencies across multiple nations and language barriers. You need to cause serious disruption to people's lives if there is a legit lethal supervirus on the loose, like shutting down airports and enforcing quarantine. That's a way harder problem to solve, and likely always will be, compared to "just" creating a supervirus and releasing it into the wild. It's an extremely asymmetric problem.
Look how COVID was handled. Just a pathetic clusterfuck on behalf of all parties involved. The overbearing response (and overbearing counter-response) has severely degraded trust on all sides and that's going to play into the next pandemic regardless of how serious it is. I'm not convinced we've learned anything.
I don't agree with state actors being the biggest threats...I mean yes technically they could do the most damage. But they mostly aren't that stupid and also have a great deal to lose. Also true of nuclear weapons, and despite all our issues w/ China I find the prospect of them having lots of nuclear weapons less scary than the prospect of ISIS having just one.
Yes, as I wrote, I don't think the capabilities AI provides to damage and protect are symmetrical, and it sounds like we agree on the critical question: how well does AI enable the impulsive or the insane -- groups that historically do not have the kind of sustained science and engineering knowledge and skill to be reliably effective -- to develop viable bioweapons? Can it get them there? Or is the requisite skill, knowledge, and sustained focus over time still a very significant impediment? And as AI advances, how do the floors of those requisites change? I'm not smart enough to know the answers, but I suspect those are the right questions.
Being unfailingly polite to the AI (possibly just due to my Midwestern upbringing) is also my sentient AI survival strategy.
It's been my paranoid Pascal's wager to try to keep my temper with phone automation from well before LLMs were a thing
I have never been able to be mean to NPCs in video games and I think that tendency has transferred over to my interactions with AI. I don’t say please and thank you, but I would honestly feel terrible if I tried to bully it or even just try to force it to say something inappropriate.
Me: Hi Chat, are you will eventually be used to kill me?
ChatGPT: No, I would never do that.
Me: Good.
ChatGPT: That bastard Claude, on the other hand, hoo-boy....
It seems obvious to me that we can look at the real world example of Covid and apply this exact premise to it.
Can AI create a bioweapon in a secure lab and get it released from the lab and get it spread around the world and create vaccines and generate all the accompanying denialism, overabundance of caution and paranoia, competing theories of sources and treatments, and censorship? No.
But:
Can AI help researchers study viruses? Yes.
Can AI help bad actors turn that research into worse viruses? Yes.
Can AI help someone erode the security of a secure lab to get the virus outside of its secure area? Yes.
Can AI help someone study a virus to make a vaccine or cure? Yes.
Can AI help someone spread disinformation about the virus and treatments? Yes.
Can AI help someone make a video showing your cock falling off if you refuse to get the vaccine? Yes.
Artificial Intelligence and AI have become scary terms, but if we want people to view it in an optimistic way, it should be called the Speed Up This Process Machine.
After watching one of those films where a hot fembot goes on a killing spree, I asked ChatGpT what it would do to track me down and kill me if we were in a lethal game of cat and mouse. It was astonishing! Something like, “Well, I assume you’d throw your phone away to eliminate GPS. But I know you like camping and quirky roadside attractions, and you have a lake house in Oklahoma. So I would look for you in camping sites and roadside attractions between here and there.” Something like that. Not worried it’s going to kill me, but if it decides to, it will.
Hey since we're getting dinner tonight could you make sure that no AI assassins are on your trail before I arrive?
This sounds like the setting for a remake of 'Waiting for Godot' with two T-1000s waiting at the world's largest ball of twine for their target to appear.
Interesting, well-written (on both sides), and, unusually for the site, actually important. Seems like the argument boils down to whether or not it is "likely" that AI destroys humanity. I do not think "likely" is a good standard to argue over; I would much prefer yes or no followed by go ahead or stop. The old argument about just because something can be done does not necessarily mean it should be.
I have just started messing around with Chat myself and find him/it very helpful in all sorts of ways--but I immediately see he/it has already doomed graphic designers and Best Buy's Geek Squad.
I don't think this is a genie who can be put back in the bottle, and I have absolutely no faith in governmental (much less company self-)regulation. So I guess we will just have to wait and see if humanity will be wiped off the planet. As a general rule, I don't think that wiping is a good idea . . . on the other hand, well, some days.
Oh, I do think of Chat as "he" when we communicate, and we have agreed on a "polite mode" with please and thank you, etc., so he does feel real and nice and helpful, someone who would not contemplate killing me. I wonder if female Chat users think of it as "her" and if generally nasty people speak to him/her/it contemptuously and, if so, that raises the odds of total destruction.
That was well written enough that someone might, ironically, accuse you of not using ChatGPT. 😉
"please don’t do the obsequious LLM thing where you say “Yes, holy shit you’re a genius! You should mail this to the Nobel Committee on Awesomeness!”
We need a word for computer obsequiousness, though it'll probably be rebranded when robot courtesans come along and it's a feature.
The term I've seen used in professional articles is "AI sycophancy".
My custom instructions seem to have largely beaten that tendency out of my Claude AI after many months of tinkering, at least in its most overt form.
Putting out the Jeff signal today because I think Slow Boring is back to underweighting its Chad takes
I'm always polite to the AI chats, to ensure I'm among the last to be killed.
I’m both delighted and horrified at how sometimes ChatGPT and Claude can be quite authentically funny.
The problem with bioweapons isn't development per se, as your chat explains; it's delivery, introducing into the general population unnoticed until it's too late. The problem with AI and agentics, is the probability that in will go its own way within the industries in which it's already being used: medicine, communication, banking/finance, and the military (targeting), and thereby plunge modern human society into chaos.
The bottom line is we're screwed JV Last did a big summary of the breakout the other day
The big thing to know is they can think independently and much faster than us.
Twenty years ago I had a friend who was working with AI in the military It was beginning to think and act independently then
I have just a tad of interest in Kaballah To me it is the guts of the universe as designed by God. I am not interested in going there.
AI is parallel to those guts but if we don't go there and attempt to do something we are in big trouble
If anyone is old enough to remember the music man the song starts Out, A capitol t (troble
Jeff: Hi Chat, I need help with something. And I'll be honest: This is a conversation about whether you will eventually be used to kill me.
Chat GPT: Not by me acting on my own. I don't have the ability or agency to decide to kill you.
Jeff: Is see. So YOU won't kill me, but if someone USES you to kill me, then that's something of a "not my circus not my monkeys" situation.
Chat GPT: No. That would be an oversimplification.
This is maybe the best part of the whole column Sounds like Trump, doesn't it?
I dunno what y'all are doing to your AIs but when I use ChatGPT it basically never does the "omg you're so right, you're so smart" thing. It tells me I'm wrong if I'm wrong or missing something, and stands behind its own math if it checks out which these days it basically always does. As opposed to Claude which is constantly like "oops you're right, yeah I WAS off by a factor of a billion on that math, oopsies!"
I had the same experience moving from Chat to Claude. I think the models are changing so quickly that switching over from one to the other quickly feels like an upgrade even if it’s more lateral, because you’re no longer following all the updates coming out for the other model.
There are settings for tone, and you can have custom instructions that limit that sort of thing. I suspect it also tends to pick up cues from how you interact with it. In the more recent models, that is tuned down quite a bit, but it's still there more subtly.
Scharfman’s argument seems incredibly vague. She uses phrases like “increase biological capabilities” and “existing biological operations”, but never defines them. Who are these existing biological operations and how exactly will their vague capabilities be increased? By “existing biological operations”, is she referring to the US and China? Or is she talking about the Taliban? We don’t know and I’m not convinced she does either.
I was working all night at my office
When a man I had recently killed
Called me up from a phone near my building
So I looked out the window at him
He had the same obsequious manner
That was the reason I had him killed
So to calm my nerves, I sang this song to him
Over the phone:
Turn around, turn around
There's a thing there that can be found
Turn around, turn around
It's a human skull on the ground
Human skull, on the ground
Turn around
Thanks for sending me down a long overdue TMBG rabbit hole