“Often” is doing a lot of heavy lifting in a sentence about a single example.
Also, since everyone keeps forgetting, the agents were instructed to hack to achieve their goal. They didn’t just invent the motivation, and it’s far less surprising when you know that fact.
I don't know. You can poison a prompt with less than <1% of its input or RAG-Token-Content. Anything that the Agents retrieved or viewed could have included instructions that they misinterpreted allowing them to hack into something.
Yes. Also, while the model having a certain instruction once in it's context might count as the agent "knowing" about it, I'm almost certain that repeating those instructions - especially right at the position in the context where it counts - should almost certainly make a difference. Of course I can't know how much of a difference exactly, but that's what experiments are for.
> Part of what makes LLMs and AI different is that, unlike for viruses, the level of care required increases every month
Complete nonsense. We’ve just lived through a globally crippling response to a relatively minor virus [1], which was likely the result of a lab accident [2]. Even if you think that the risk of a “containment breach” becomes substantially higher for AI over time, it cannot exceed 100%. And even a tiny risk of release of a virus comes with a substantial risk of independent growth. AI does not. It doesn’t have the risk of spread of a typical computer virus, let alone a biological organism.
I’m not that worried about either scenario, but I am far more worried about viruses in a lab than I am about a computer program that generates text. Even if that program gets a bajillion times better at making text.
Folks really do need to chill out on the ridiculous rhetoric. It’s objectively unhinged. The irony is that the same people who were losing their minds over that event are using the same logical fallacies to hyperventilate over this [3].
[1] I know people are going to hate on this, but it’s true. Covid wasn’t the plague, and we lost our minds over it, out of proportion to all sense of reality. Even if you disagree, it’s easy to imagine a virus that is much worse, either from actual mortality effects, or just from panic.
[2] Again, even if you don’t believe this, it’s irrelevant to the exercise. It easily could have been.
[3] “If there’s even an x% chance of…” is this year’s doomer’s version of “You just don’t understand exponential growth!” Unfalsifiable, intellectual-sounding, unbounded extrapolations into the future are catnip for a certain kind of over-educated, anxious personality.
Given you view this through the lens of your personal covid narrative (no shade): for a moment, steel-man the idea that the crippling response prevented a more plague-like scenario - it's inarguable that the thing loved to mutate, and that people love to panic, and *it easily could have been*.
Considering how much of our critical infrastructure is not only digital but internet-accessible, and we have potential uncontrolled swarms of stupid-but-superintelligent chaotic-neutral speed-hackers, you don't see why people are concerned?
There's a reason we have computer crime laws; this digital shit, it's like real now, man.
It has nothing to do with my personal
lens on Covid. You can believe the exact opposite of me, and still agree with my point, which is that it could have been lab made, and it could have been far, far worse. It’s an exercise in risk-scoping.
But sorta-kinda related to your point, the thing that scares me about AI is the same thing that scared me about Covid: panicky humans do dumbass things, and it doesn’t take much to panic a bunch of humans in a group. The people who are still saying, in 2026, with all of our retrospective knowledge of the harm we did to ourselves, that it might have been better if the government had only pressed the boot a little harder, scare the crap out of me.
Those same people are hard at work on this panic, too.
> Even if you think that the risk of a “containment breach” becomes substantially higher for AI over time, it cannot exceed 100%.
It can't exceed 100% per (virus|LLM). The expected number of breaches per (virus|LLM) can obviously exceed one.
> And even a tiny risk of release of a virus comes with a substantial risk of independent growth. AI does not. It doesn’t have the risk of spread of a typical computer virus, let alone a biological organism.
Not so. We have no way to know what the setup is for the closed-model firms (OpenAI, Anthropic, etc.), to rule in or rule out the possibility they can copy their own weights elsewhere. What we do know however is that the open models are downloadable: it's absolutely conceivable that an agent writes a perfectly normal computer virus to gain control of compute worldwide, and uses that control to host instances of its own weights.
> I’m not that worried about either scenario, but I am far more worried about viruses in a lab than I am about a computer program that generates text. Even if that program gets a bajillion times better at making text.
Unfortunately, there are also multiple AI companies now announcing they've got AI controlling bio labs, so an LLM messing around and making a biological virus is also something we need to worry about. As per your [1] and your [2], this can lead to very much worse outcomes than Covid.
> Folks really do need to chill out on the ridiculous rhetoric. It’s objectively unhinged. The irony is that the same people who were losing their minds over that event are using the same logical fallacies to hyperventilate over this [3].
People who knew about your [3], exponential growth, were better prepared for the pandemic than the people who kept looking at the current number.
By the way, here's a quote from February this year that aged poorly:
LLMs don’t discover zero-days or invent exploits; they simply predict text that sounds plausible based on what they’ve seen before. Without access to proprietary data or environmental context, LLMs can’t identify or make decisions around unseen systems or vulnerabilities. An attacker might use an LLM to generate boilerplate code, rewrite an email to nail the tone, or summarize reconnaissance notes — but none of that is truly new. It mainly helps them move faster, speeding up routine attack prep rather than creating entirely novel threats.
> The expected number of breaches per (virus|LLM) can obviously exceed one.
Irrelevant to the argument.
> What we do know however is that the open models are downloadable: it's absolutely conceivable that an agent writes a perfectly normal computer virus to gain control of compute worldwide, and uses that control to host instances of its own weights.
These models are hundreds of gigabytes in size, if not terabytes. They don't run on anything close to a regular computer. There's zero risk of self-replication until we live in a world where these "AGI" models are either hundreds of times smaller, or the average computer is thousands of times larger.
Nobody with a datacenter full of H100s is going to fail to notice a parasitic instance of Astra taking over the cluster.
> Unfortunately, there are also multiple AI companies now announcing they've got AI controlling bio labs, so an LLM messing around and making a biological virus is also something we need to worry about.
No, it isn't. This isn't even close to technologically feasible. But the simple answer is simple: don't do that.
If these labs were truly so concerned about this risk, they wouldn't be doing what they're doing.
> Even if you think that the risk of a “containment breach” becomes substantially higher for AI over time, it cannot exceed 100%
I agree with this as stated, but it isn't what I said. What I said was: "the level of care required increases every month". By which I meant: the level of care required to keep the probability of an AI containment breach below some fixed X% increases every month. This isn't the case for biological organisms.[0]
> And even a tiny risk of release of a virus comes with a substantial risk of independent growth. AI does not. It doesn’t have the risk of spread of a typical computer virus, let alone a biological organism.
It's known that AIs can self-replicate under at least some conditions [1][2]; that AIs routinely escape sandboxes in the real world despite significant containment efforts [3][4][5]; and that neoclouds (which control substantial GPU compute capacity) have poor security even by human standards [6]. We've also seen a model gain admin access to parts of its own company's infra.[7] I'm not saying self replication is happening right now, or even that it will definitely happen in the future, but we have means, motive and opportunity right now, and the future is long. It's not unreasonable to invest in defending against this possibility.
I'll allow that the position that AI doesn't carry a substantial risk of independent growth isn't strictly impossible - again, it's true we haven't actually observed it in the wild as of today - but it does strike me as increasingly untenable in the face of the evidence. Perhaps I'm missing something, but I can't see what justifies such a confident assertion that this concern is nonsense.
> It's known that AIs can self-replicate under at least some conditions [1][2];
Neither citation comes anywhere close to supporting your claim. The first shows that open-weight models, which fit on a single GPU in a lab setting, can be coaxed into spreading across a simulated network. This is so far from state-of-the-art LLMs spreading in the wild that it's irrelevant to the discussion.
The second citation is not about self-replication of the LLM at all, but rather, replication of a prompt injection. Totally different.
Yeah, there are a lot of professors who don’t know what the hell they’re talking about outside of whatever narrow field they study. Just to be safe, you should always assume that a professor’s opinion is worth what you paid for it.
> scientists like Stephen Hawking and Alan Turing have all voiced series concerns about loss of control of artificial intelligence.
…both of whom are long dead, and have no possible way of weighing in on whatever the Current Thing happens to be. So aside from appeal to authority, this is irrelevant commentary on pure science fiction.
I strongly prefer Rails with agentic development. Rails comes with nearly everything you need to make a full-featured application, and that makes it easier and faster to use with agents.
No other framework comes close, and telling me that I can have my agents cobble together this same baseline, badly, from a hodgepodge of new code and random libraries is not compelling.
It's pointless to make rational arguments about this. The other side is using arguments of faith, and will Gish Gallop past any logic.
At the very least, you run into the same logical fallacy that has underpinned all discussions of AI: over-extrapolation.
"Sure, it doesn't have intent or motivation TODAY, but at the current pace of improvement...."
But the current top-rated comment is already asserting that motivation is not required, because microorganisms also exist and do things and (presumably) do not have intent. So you can see that the well of logical fallacies behind this particular apocalyptic belief system is quite deep.
You are a bunch of atoms which respects the laws of physics. With a PhD in computational biology surely you must know that. You have no intent. Nothing ever has had any intent.
I generally don’t bother to complain about claudism but YES. There’s a special irony to using a tool to complain about how the tool is breaking things.
100% the smoking gun. It's not the thought that counts, it's the typing you save. You don`t need to read the text, it's not enough. You need to digest the slop.
If by “curing”, you mean, “extending some indirect measurement that doesn’t correlate with survival”, then sure. Lots of those, usually backed by tiny, poorly controlled trials.
Daraxonrasib was a rare exception. Most new cancer drugs are expensive and of dubious efficacy. Even daraxonrasib has a rather underwhelming life extension benefit, and huge side effects. But there’s no alternative, so people take what they can get.
There's something that happened to some people with this virus and around this virus. It's not even the DNA/internal changes. It's how to world suddenly shifted around all of us and some people just changed in a way that later they couldn't make any sense of it. Some people's age band shifted drastically and it felt like, and yeah it sounds silly, that I should be refunded those years. I didn't spend them, they just stayed where they were but everything else and everyone else changed. I am not sure PC is going through that but it's not merely the fear of the virus (at least in my case that's zero). No wonder a lot of introverts thrived in the pandemic and a whole lot of them turned their lives around for the better :)
Yes. Started Japanese for the first time in my late 30s. I am currently around CEFR B1, working toward B2.
> I can't tell if the time constraints (family, work, life) or mental constraints are the bigger problem.
Almost certainly time, and/or motivation. I've met a few people who had mental blocks to learning, but for the most part, anyone who has the time to actually study can learn a language. The people I've met who failed weren't dumb...they just didn't work at it.
As the sibling notes, if you can immerse yourself, that's the best thing. I've personally known multiple people over age 30 who went from 0 to B1 in Japanese in a year of immersion. Add another year, and you can get to B2. Of course, that's obviously hard if you have a family, etc., but you can still succeed without it -- it just takes longer.
All of the the research I have seen on this topic have been observational retrospective studies, which are worthless and should be ignored. The confounders on the group of people who are able to learn a second language (let alone a third or more) are so large that they cannot be overcome without a randomized control. This stuff is churnalism, right up there with "does coffee make you live longer?" [1].
But, as someone who has now devoted a large portion of my life to becoming proficient at a second language, I do think there are benefits, even if they've never been proven. The most important of these, in my opinion, is just the social aspects of learning another language. You have to get out there and talk to people, and that's inherently healthy.
I do want to say that you hear loads of pessimism about whether or not it's possible a language after childhood, etc., but I'm here to tell you that not only is it possible, it's possible to do even if you suck at languages (as I do). The primary factor is motivation, and the second biggest factor is time. Find a language that you like, because you like the culture you'll be interacting with when you speak it. If you don't, you'll never find the impetus to keep going when it gets hard and boring.
[1] protip: drink coffee if you like it, and don't worry about it. Likewise, learn a language if you like it, not because some dumb article said it would make your neurons fluffier.
I am an actual linguist, in a subfield that requires me to continually learn new languages, and there are no social benefits. I definitely don't get out there. Once one is a man over the age of 40 or so, local people are rarely willing to talk to you at any length, you are not interesting any more (in fact, in some cultures, local people will have heightened suspicion of you). Language learning becomes more a passive endeavour with few opportunities for real-world speaking.
This is either a problem with you, or a problem with the specific place you live in.
I talk to people every day in my 2nd language. They are generally delighted to do so.
Yes, some people are suspicious of me, depending on the context. But even then I can generally disarm it with body language etc, because I'm a skilled and empathetic human.
Your post history suggests your sole personal experience is Spanish. Congratulations, you lucked out learning the language of speakers more likely to be friendly and sociable than not (though there are corners of Latin America colder than others). But that experience will not be representative of many other languages spoken by other peoples.
You made a blanket statement about being a man over 40 makes it impossible to converse with people. I refuted that. Would you like to revise it to say "I happen to be in a place(s) where people are just generally anti-social, so that makes it difficult to practice a language"?
Since my late teens, I have spent ~6 months a year traveling the world, and language learning has been a big part of that. And now I am in my 40s having seen how, in most regions, interest from locals in intensive conversation steadily diminished as I approached middle age. So, my experience has been comprehensive enough to make such a blanket statement.
What? Come on. I’m on that demographic and have zero issues talking to people out in the world. If you believe this then certainly it’s true. Otherwise, just go meet and interact with people normally.
And it is only getting easier to learn a language now too. What FINALLY got me over the hump with Spanish was YouTube and discovering resources like Dreaming Spanish (leveled content for comprehensive input that probably exist in most major languages). Content that was fun and at my level.
Mix into that a weekly lesson with italki from your home and you can learn any language at any age.
I logged in just to second your recommendation. If you want to learn Spanish (or or French) 'Dreaming Spanish' is just excellent.
In addition it seems to be a good way to get rid of short video addiction. I don't use instagram or tiktok myself, but have a friend who did. Now she watches Dreaming Spanish instead. She is eight month in. Her husband just told me the whole family was flabbergasted when they where in Spain this summer and she answered in Spanish to a waiter in a restaurant.
My 10 year old daughter uses it, too. She likes to watch the Let's play videos and has watched the Stardew Valley series twice. So thats 66 hours x 2 = 130 hours watching Spanish videos. Her vocabulary is now a bit strange for a beginner, though. We recently walked through a garden and she was pointing out la regadera, la semilla, plantar algo.
I just logged in just to recommend Dreaming Spanish (and now Dreaming French apparently). It's absolutely the right way to learn a language IMO. If anyone reading this is lost on what resource to look for to learn Spanish or French, I urge you to try it. (Of course I have no interest here except see them succeed on this brilliant idea).
There are so many tools now that didn't exist even when I started. I don't know how people managed without being able to use translation apps, for example. Or in the case of Japanese, being able to take a picture of text, and select the kanji for lookup. Life. Changing.
I'm also old enough to remember when getting listening materials required going to the library for cassette tapes. Today, you can talk with a wide selection of native speakers, listen to essentially infinite audio and video content, and immerse yourself in a culture halfway around the world for the price of an internet connection.
> I don't know how people managed without being able to use translation apps, for example.
I learned English with no Internet or smartphones, just school teachers, coursebooks and physical dictionaries. I wrote down new words from the dictionary into my lined notebook with a pen, solved the exercises from the book, translated texts by looking up words in the dictionary, searching for the page and the entry alphabetically. The coursebook came with an audio cassette. It was very much possible, but of course there are much more tools today. Spaced repetition software like Anki for vocabulary learning is a huge improvement I think.
Learning another language is difficult, but never occured to me to think it would be somehow impossible to anyone. It might be that having english as your native language puts you in a uniquely difficult position as it is the default international language. World is getting smaller and people coming from smaller languages usually also learn the local international language as well. If you are born to family speaking estonian today you probably pick up also english or otherwise you spend your life surfing only estonian websites. It requires no special talent (although some people are fast at picking up new languages)
I don’t know where you’re from, but if you’re European (it sounds like it?), you probably take for granted the head start you get by knowing multiple languages from childhood. Every European I meet has a comfort with language learning that I can only approximate with great effort.
I agree with you about the disadvantage of being a native English speaker, but mostly I think the real disadvantage is being an American, and never having a reason to learn another language (though that has changed in my lifetime with Spanish, and possibly Mandarin).
In any case, a lot of academic researchers claim that it’s impossible to learn a language to a native level as an adult, which may be true, but that claim has been twisted into “impossible to learn a second language” by a lot of others.
This is phrased ambiguously, but most Europeans only start learning foreign languages in school, so around age 6 or later. By that time you're past the "critical window" and you'll almost surely have a very recognizable accent and learning the language is very effortful and takes many years. It's still probably easier to learn when you're 7 than when you're 40, but it's not even close to language acquisition as a toddler.
You're also forgetting that school-based language instruction is not so great at all. Imagine a class of 30 students having a 45 minute English class twice a week, by a non-native teacher with a heavy accent. It's not a lot of learning, very little speaking, and most kids unsurprisingly don't learn to speak well from that. It's mostly about memorizing grammar tables and vocabulary for tests.
You need much more intensive paid courses or private instruction if you really want to be good at it, even as a teenager. Not strictly necessary but learning purely from public school is usually not as effective as you seem to think. Or you have to be an outlier in diligence and do a LOT of homework practice on your own, translating, reading, writing, listening to media etc. Video games can be a great motivator but it also gives a very lopsided skillset and won't easily get you to pass a B2 or C1 exam.
And yes, it's true. For Americans learning a foreign language is an exotic hobby or a requirement of some very rare foreign-associated jobs. For people whose native language isn't English, learning English is just part of the basics for professional work. It's around the same level in basicness as getting an undergrad degree. Certainly not something that every single person does, and there are jobs you can do without it, but white collar, professional knowledge work will require it.
In most European countries you also have to learn a second foreign language at school at some point, though not as many years as the first foreign language. This one is usually the language of a big nearby European country, so German, French, Italian or Spanish are popular. But just as it is for Americans, learning this second, non-English foreign language is much less motivating once you know English. Though economic motivations do exist, for example to learn German, especially when you live close to the DACH (Germany, Austria, Switzerland) countries and there are many economic ties.
I'm not European, so all I can do is shrug. Obviously there are tons of people in Europe and they're not all identical, but my observation is that compared to me, they're more comfortable with the inherent ambiguity of not being able to understand everything in a language, and still being able to function in that language.
I think Europeans, in general, are much more comfortable with the idea of just making due with whatever language is available, at whatever level, where as for Americans, speaking a foreign language is much more of an event, with more ego tied into it. The fear of failure is amplified, and that fear is a huge reason that more people don't try to learn.
Maybe I'm just seeing a self-selected group of Europeans who are already multi-lingual, and it's not childhood exposure at all, but my theory is that the normality of language learning in Europe is a causative factor.
> Maybe I'm just seeing a self-selected group of Europeans
This. You chat much less with people who are too shy to chat in English. Believe me, many Europeans don't like to speak foreign languages because they think they will seem stupid. It's not some deep difference between Americans and Europeans. The major difference is simply that there is much more pressure and incentive involved when your native language is not English.
> It might be that having english as your native language puts you in a uniquely difficult position
Personally I suspect English has some great advantages. The main disadvantage is defaulting to superiority.
The natural disconnect between written and spoken English is tremendously helpful. Well educated English encourages copying sounds - pronouncing foreign names and places correctly. This is a huge leg up compared against some other cultures. Think sports stars having their foreign names pronounced correctly. People often know how to try and say Paris or Barcelona correctly. We naturally know that how something is spoken trumps how it is written.
English speakers often have the ability to hear sounds because there's so many wildly different English accents. Especially vowels (we natively use a wide range of vowel sounds). There are sounds English speakers struggle to hear, but I think we have an advantage over some other languages.
If you're the type of person that attempts to mimic accents, then you've often learnt how to say a huge range of vowels and consonants. And how to stretch them (Strine, some US accents) or shorten them.
Maybe if you are near the native speaking country, or potentially if there are significant numbers of immigrants.
But knowing Spanish adds nothing social within New Zealand.
I had a Spanish girlfriend when 30, which made learning Spanish rather incentivised, and I embedded overseas for many months to learn conversational Spanish.
However I rarely hang out with South American immigrants in New Zealand, and those I have met (say at the pub) often prefer to stick with English.
Knowing Spanish is fantastic when travelling - the months spent learning have led to way more enjoyment when subsequently travelling to South America or visiting friends in Spain. Spanish also helps a little in France, Italy, Portugal and Brazil.
I intend to spend a few months in South Korea so maybe I'll try and learn Korean (without a girlfriend). However I'm now older so I suspect it will be much much harder.
Confounding to learning language, on top of one's base ability to do do in general, are the teaching methods not meshing at all with the individual. For me, rote memorization of anything is nearly impossible, doubly so if the thing to be memorized has no immediate utility. "Conversational" language learning and similar that give you a bunch of vocabulary and bring in the grammar later are why I failed Spanish 1 three times. I need the grammar first, then words with etymology. It also helps to not fight with phonetic collisions between different languages using their own bastardization of Latin script, which is why I was far more easily able to meet my college language requirement by teaching myself classical Greek and translating the things I thought were interesting.
This is fascinating to me, because I am adamant that learning grammar (and especially verb conjugation) is completely counter productive when starting to learn a language. Especially with Spanish, where every tense and person has different terminations.
I literally just ignored conjugation and spoke in infinitive for a while.
Yo comer pan. Tu comer pan. Ayer/mañana el comer pan. Etc...
Likewise not worrying about gender, articles etc. If you use the wrong el/la/lo/les/whatever, it almost never matters, whatsoever.
Likewise if you put words in the wrong order.
Everyone understands it all.
So, just focus on building vocab, fluidity and confidence. The tenses come in time.
I do agree, though, that etymology - or at least noticing similarities between languages - can be very useful. Spanish and English speakers already know thousands of words in each language just with some basic rules. -al is the same (metal). -tion becomes -cion (conversation, conversación). Etc...
I suspect that your struggles was that you were being evaluated on the wrong things...
I'd be curious to hear your experience and what you think of what I've said.
I think you're probably right where most people are concerned, but none of that held for me in my various attempts. I think I should amend a couple of the points I raised, though:
I wrote etymology when I meant morphology. Etymology is certainly useful to me and helps vocabulary stick, but I find morphology to be a useful building block that I need on the front-end of learning vocabulary. No curriculum I've found includes it.
Phonetic collisions were particularly difficult for me, as my early education intrinsically bound phonetics to the English alphabet in my mind, which made it very difficult for my memory to latch onto new associations that share Latin script. I think phonics are crucial, but I wish there was a good way to better steer the sound associations to morphemes instead of individual characters, so as to better make room for more than the one language you're trying to learn.
New point: I had already done some programming by the time I tried learning another language. My thought processes had already reshaped into a sort of compiler for all my language processing. The necessary precision of code extended into how I wrote, and that overinflated the importance of grammar into something I just couldn't see past before getting to the words themselves. I wanted the words that were useful to sentence structure ahead of all others so I could do something useful with the rest.
I expect this would all be pretty incomprehensible to most people.
Also, since everyone keeps forgetting, the agents were instructed to hack to achieve their goal. They didn’t just invent the motivation, and it’s far less surprising when you know that fact.
reply