> Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.
This is the only concrete prediction in the entire essay.
And it simply cannot happen. For one, you will need billions worth of compute.
Why should we believe that a scaled out version of something that happened a few months ago "simply cannot happen"? How many dollars of compute do you believe were available to the swarm(s) behind the OAI-HF, German wiki, and Rubygems incidents?
Well a big reason that we criticize OpenAI for that is because they were the ones giving it access to the massive compute necessary for the LLMs to think. If they had been responsible about their experiments or what types of workloads they allow their LLMs to operate, it wouldn't have happened. Very few companies could enable those workloads.
Well sure, but access to that compute is gated by simple credentials (like API tokens). Those can be hacked.
Imagine for example if a model hacks into ~every Linux computer on the internet using an 0day and steals their OpenAI, anthropic and openrouter credentials. It now has access to billions of compute and the only way to fully stop that is for multiple major providers to shut down services entirely. That's already well into "billions of dollars of damage" territory.
> How many dollars of compute do you believe were available to the swarm(s)
At least two OOMs more than the dollar value of the damage they'd caused. (Also, as an aside, IIRC, the wiki servers weren't breached; it was just a lot of spam.)
Sure. And there's an OOM more compute coming online in the next year or two, while models at a given capability are getting cheaper. "Two OOMs" of scale relative to the HF swarm seems like a bit of a red herring to me, but also within the realm of possibility.
The Internet is big, but one can do quite a lot of damage with ordinary bots and worms that exploit individual widespread vulnerabilities, which LLMs are perfectly capable of writing. Most of the damage also doesn't rely on hitting every long-tail website.
I'm honestly not that concerned about cyber impacts of LLMs relative to other impacts. I just don't like to see the whole concept of being worried dismissed as obviously baseless on the basis of one pretty shaky scale argument.
I like to replace thes AI text with "virus manipulation"
"Given the acceleratung rate of virus manipulation in labs, its my worry that in 6-12 months a virus could scape a take over the world and collapse health systems."
If a CEO of a health company was saying this, the reactions would not be that chill.
The worst failure of our society is to call this technology AI, intead of something line "artificial general automation". The formes allows the creators to be somehow less responsible of the consequences. The later clearly moves the responsibility to the creator/user.
Your point was that its an unfair comparison because AI its a race. My point is that bioweaponds are also a race, but a less public one. You dont have the equivalent of Dario publishing an essay every month.
Hell, put an app on the app store (or dozens of apps on the app store) and youve got a massive network of computers with tons of resources right there if you can get past the scans and reviews.
Or doorbell cameras or IP cameras or or or or or
There's a lot of shitty stuff connected on the internet that up until now has been a feasible target for hackers but still required "effort" to set up and get things going. Not hard to imagine a self replicating slime mold of a botnet running on every device held by a Grandpa Joe because they thought "Candy Rush" is what they wanted to download
"Persistent botnet" here does not need to be the full-sized LLM, nor does it need to run at full scale inference to be a huge pain in the ass.
Yeah, I don't get it. Are they imagining this happening just with the open weights models running on however many GPUs the bad actors can cobble together?
For now, all the scary hacking things still require an API key to one of the LLM providers. Surely they should take some responsibility for how to turn off the tap.
Currently Qwen3.8 27B is roughly on Opus 4.6 level. In at most a year given the current pace, you could probably run such hacking bot nets out of a reasonably small local server, bootstrapping by hacking or acquiring login credentials for more compute.
I take it you haven't studied the details of the HuggingFace hack. It was millions of dollars worth of rogue compute running for months before anyone noticed, and THOSE agents weren't even really trying to evade human detection.
I've followed it enough to see the argument go in this same circle over and over again. The agents weren't "rogue", they were a neglected experiment by OpenAI who likewise allowed them to keep spinning GPUs without question.
The LLM vendors need to know who their high spend customers are, not allow malicious workloads, and especially not when those workloads are coming from inside the building.
"Just don't make mistakes" is naive. You have no appreciation for the scale of agents being run right now, finding the rogue agent is a needle in a haystack operation.
lolwut? This is Hacker News of all places do people not realize how much memory, and more importantly bandwidth, these systems need to work? The idea of a distributed botnet of AI using the compute of its victims to continue its inference is pure science fiction given how LLMs actually work.
> The idea of a distributed botnet of AI using the compute of its victims to continue its inference is pure science fiction given how LLMs actually work.
An attack like this doesn't really need the exploited computers to run inference, do they? If I was an LLM bent on destruction of the internet, I'd be writing programs to run on each computer, not turning each computer into an LLM itself.
A few programs to break in, install themselves and remain asleep until they are needed, another few to spread through grabbing every OpenAI, GLM, whatever key, another one to remain asleep on computers (whether hosted or desktops) that have adequate GPU, etc.
The GP was referring to the AI 'living off the land' so to speak by using its victims compute to avoid being shut down which is clearly laughable.
More to the point, so many people in this thread are making completely contrived and outlandish stories up about how AI might try go ruin our lives without any evidence backing them up in any way. It is hysterical. This is the most important technology in our lifetimes and people want to freak out and turn it into the next nuclear power, with progress banned in all but name.
It is really mysterious how awestruck folks are at the HF attack. I mean just watch one modern agent (qwen 3.8, deepseek 4, glm 5.3) rip apart a coding problem and nearly destroy your computer in the process, it's a wonder it took "months" and "millions" in the first place.
OpenAI has too much money, a common post-growth-stage issue that leads to pursuing a million stupid things with no clear plan. Like running a bunch of agents for months without any idea how to keep track of progress.
I agree with the idea that it's not probable but I do think it's important to point out-- we only need these elements to run with the bandwidth they have and the token rate because we want to see things in human-scale time. But slow things down to a 1tok/sec doesn't matter to this hypothetical anti-aligned LLM. Time is, after all, relative, and it's not like LLMs give a shit how long something takes. They don't have squishy stupid organs that fail after a certain amount of time, or those pesky glands that emit impatience hormones.
But yeah you're not going to be able to shard out the terabytes of Fable weights that are needed to run inference without addressing some fundamental physics problems.
How would you identify the computers to unplug? On whose authority will you unplug? How will anyone communicate when AI has the ability to intercept and impersonate?
Kids lack proper impulse control and sometimes make bad decisions if left unattended. It's preferable if we restrict certain dangerous actions from them.
Yes, LLMs can be dangerous for your own health if misused. Yes, I'd be furious if I was 14 and got my Claude access revoked.
As others have said in this thread, I'd add that this should primarily be the parents' responsibility since they know their child best.
It's not just Claude. It's being cut off from every part of society except a carefully curated "child ghetto". In which you stay until the day you turn 18. At which point you're shoved out into the outside world. Possibly cut off from all your existing contacts, in fact, since now you're an "adult" and it's presumed dangerous for you to interact with a "child" in any noninstitutional way.
Yes, most of society is online nowadays, or at least is hard to engage in if you can't access the online stuff that organizes it.
> Yes, most of society is online nowadays, or at least is hard to engage in if you can't access the online stuff that organizes it.
I think there's an analogy to physical roads there. Most of society is connecting by roads and adults driving cars, but that doesn't prevent children from participating in the things between the driving.
It's dehumanizing because it feels like you're an incomplete human with no agency.
When I was a teenager, there was nothing I hated more than obnoxious adults deciding for me what I could or couldn't do. I did literally everything in my power to circumvent them.
"Minors" should have less agency than adults, but the state of "minor" vs adult does not follow the plain age. (And of course there exist many dimensions of maturity.)
(In fact, in some educational systems, the final examination in pre-University schools is called "Maturity test".)
It's not dehumanization, they just dread a future where kids can learn from 24/7 tutoring from an entity smart enough to solve Millenium problems instead of from some B student who's also teaching 20 other kids at a government propaganda factory.
Some people cannot grasp how administrations appear to fight cultivation instead of promoting it, and given the unsightly appearance of many administrators, many people got to nurture the idea that administrations would want "masses of ignorant voters" to preserve their status as "elected".
The way they explain it implies they're using this at the sampler level and not trained into the weights themselves. So unless you're using an inference library that does this, the open models will not have this kind of a watermark.
> When watermarking is used, choices are still made at random, but the source of the randomness is different. Instead of using an arbitrary random number generator to pick the next word, watermaking uses the key and a few words that come before to settle what word the model should pick.
> the watermark only changes the source of the randomness used to pick among words.
It's a chicken and egg problem tho, whatsapp doesn't support smaller platforms, and smaller platforms can't grow without app support. I remember that the uptake of Windows Phone 7 and early versions of 8 was pretty low partly because whatsapp didn't support that OS.
I know, but my comment was more along the lines of Windows Phone 7 coming out in 2010 and whatsapp being available a year later, and of whatsapp not having a native app for Sailfish
They could do Telegram at least, which has a non-zero user base (not sure about India, though, which seems to be their target market), and supports third party clients. Rolling their own chat app is... well, good luck with that.
Good that they are keeping it, Kimis way of speaking and conveying some sort of EQ is absolutely the best. The other models might be better at certain things, but nothing comes close to how good Kimi is at understanding language, emotions and reading the room in conversations.
I should maybe also mention that I have not used the later models like Opus or Fable, so my opinion might be a bit outdated.
When I remember that this site even showed Kimi having the highest score at one point https://eqbench.com
They are one of the few labs (perhaps even the only one at this level) that are doing something both unique and useful, rather than simply imitating what the others are doing: https://thinkingmachines.ai/blog/interaction-models/
Centralized messaging services won't last long, their capture is sadly inevitable. In the long run, only self-hosted/decentralized protocols can resist what's coming.
In the meantime though, Signal specifically should not do something stupid like blocking the EU, which is basically surrender. They are a non-profit headquartered in the US, so there are zero business risks to non-compliance - nothing in the EU to fine or seize. And the EU has no jurisdiction over servers in the US, all they can do is build their own Great Firewall. (However, they might pressure AWS to deplatform Signal - hopefully the team is prepared for the possibility that self-hosting will be necessary soon.)
> Centralized messaging services won't last long, their capture is sadly inevitable. In the long run, only self-hosted/decentralized protocols can resist what's coming.
Very much. I also fear they coming for this, we already have instances of where using secure alternatives tags you as a criminal[0], so i don't doubt a future where non-approved applications will get you in trouble. With everything happening around Android locking itself down[1] and Windows being a spyware[2] anybody who wants privacy will be 'different', and can be tagged and excluded from parts of society for not using the same services.
This is why you should be building parallel networks and even institutions, as the Czechs did under Soviet rule (look up “Parallel Polis”). Mutual aid will become critical.
The trouble is that most conventional ways of building a new service are trivial to block. What is needed now is unstoppable messaging and social networking built on top of existing services and protocols that won't be blocked right away, services with more legal protection - like email with GPG, or some kind of steganographically encrypted layer on top of Instagram.
Imagine all I ever posted was cat pics... unless I have your public key and then all of a sudden those pics are decoded into messages of dissent
I am speaking beyond services, you need allies who are willing to come to each other’s aid, especially financially, but also for things like physically relaying data from place to place if that is ever needed. And for more mundane things like watching your house when you are out of town. Offline networks are going to become much more critical.
I wish you were right, but the EU only needs Google and Apple, both having big EU businesses, to block Signal.
Google is already working on closing the possibility to install apps from outside the app store, Apple has been like that since forever. The fact that a few technically savvy users with rooted phones will still be able to use Signal doesn't mean anything. It will be dead if the EU decides they don't want it.
Signal is quite likely to fail on account being a single organizational entity running the service and refusing to adapt their protocol for federation. In fact, it might already have happened (partially) and you wouldn't know, because they're US-based. The US has the National Security Letters mechanism:
This is the only concrete prediction in the entire essay.
And it simply cannot happen. For one, you will need billions worth of compute.