While I have my reservations about Amodei and his company, I'm nevertheless a happy user of their software. And I'm in agreement with him (and Sanders) that we should all. slow. down.
To my mind, the last great arms race between nation states was a misdirected love triangle between USA, Russia, and The Bomb, and look at all the damage that did.
Since AI is the new arms race between USA and China, slowing down may just give the humans involved enough time to realize they should be loving one another, instead of the machines.
Maybe saying, "let's slow down", is another way of saying, "I love you."
Or, maybe it's just: "let's not all of humanity kill ourselves like some bad ending to a Shakespearean tragedy."
Either way, it's a better note than, "We must achieve sea/air/nuclear/quantum/AI/spiritual supremacy before those other bastards do!"
Impossible to prove hypothesis: nuclear weapons prevented a world war.
Facing the facts about nuclear weapons means owning the good (probably prevented wars) and the bad (at the very least there were severe environmental and economic consequences).
This is not mathematical logic. Of course you can't prove the causality of anything in history. The only way to do that would be to invent a time machine, delete the nukes and watch things play out.
There are direct quotes from Nixon and Reagan that claims this to be true. That is enough evidence for me.
The Cold War was the best possible outcome. The alternatives (super power hot war, nuclear armageddon) did not happen. Nuclear weapon potential was a large part of that, as was direct lines of communication. Unfortunately because the later is no longer a thing, and our president declined to extend the existing agreement (a problematic pattern), we now again face proliferation of nuclear weapons.
Some people believe WW2 started many years earlier, but most historians don't put the start date at the conflicts going on before the war became intercontinental with sides working together on a global scale.
War has still been going on since 1945, much earlier if not always, so akshually WW2 never ended or is just how it's always been? (mostly in places that are not "first world" / The West)
There's a simple and very logical definition of what a "world war" is - it's when international law completely breaks down, and the only way to get back to a semblance of order in international relations is to have the victors of the ensuing shitstorm impose a new set of rules in some sort of grand finale pact.
1945 was clearly that. 1920 with the League of Nations attempted to be that, but failed.
Whatever happens after the current turbulence we can be sure that there won't be a UN and NATO and a World Bank at the end.
P.S. The first modern world war was the Thirty Year's War.
Because the UN, NATO, etc., are clearly not working now. They're not gonna be fixed to work as designed because that's quite impossible in today's world.
That's been a posited theory for decades - it's not just some HN edgelord thing. There are some obvious problems with it of course and the P5 have instead taken a maximalist attitude towards keeping the genie in the bottle (notwithstanding Atoms for Peace and accommodating India/Pakistan when it became inevitable).
It's weird to me that seemingly both sides are taking opposite positions to their philosophy.
Open Source AI democratizes the means of production to anyone with a computer. And yet, the hyper capitalists are defending it, and the progressives think it should be exclusively in the hands of 1-2 large corporations.
The problem is that at the business level and at least for a large part of the US government, we need to assume that there are no 'good' actors.
Anthropic needs regulation in order to prevent AI from becoming a commodity. This of course does not benefit all tech businesses equally, especially those that are not currently at the AI frontier. So when JD Vance talks about AI, he talks using the mouth of Peter Thiel who may not see benefit from the same policy as Altman or Amodei.
The rest is just public support posturing and most of that is bullshit meant to distract from the high rollers game of winners and losers. The philosophy is money and power, who gets it and who doesn't. Us normies aren't really participants in the game, except where we are being manipulated into cheering for one side or another, and with little stake in the outcomes (although selfishly, I'd be pissed if I didn't have open models to tinker with).
Open source is fundamentally a vehicle for commoditization. This is great if your business is not AI and your business is instead something like GPU hardware or some product that uses AI. But it means that eventually, selling AI is not going to be the money maker.
OSI proliferated open source on a business strategy called "commoditizing your complements". These big companies don't do it out of benevolence. It was pitched to them in a way that FSF did not (which was more about morals and ethics, something business care little about), and it caught on. And the software business became about ads, consulting and cloud services instead.
> was a misdirected love triangle between USA, Russia, and The Bomb, and look at all the damage that did.
Can you be specific about the damage? We currently live in the most prosperous times on earth for humans. I'm not sure what you mean by damage.
Nobody can explain why an LLM can be so capable as to be able to wipe out humanity and pose a greater threat than nuclear bombs but not be so capable as to be able to protect humanity against that threat. Are we just handwaving this with "entropy"?
> Maybe saying, "let's slow down", is another way of saying, "I love you." Or, maybe it's just: "let's not all of humanity kill ourselves like some bad ending to a Shakespearean tragedy."
Okay nevermind, I think it's pretty clear you just want to wax poetic about all of this.
> Nobody can explain why an LLM can be so capable as to be able to wipe out humanity and pose a greater threat than nuclear bombs but not be so capable as to be able to protect humanity against that threat.
It is absolutely explained (for those who actually care about reading). Simply put, AIs are working more and more like blackboxes - there's no guarantee that an AI of the future will be aligned, or if it will be faking alignment. This is not speculation - alignment faking has been observed in experiments. This is exactly why Astra's developments have been worrying (in principle).
And bear in mind that recursive AI development started already to be a thing. Which means: inner misalignment may trickle down the generations, and humans won't detect it.
Having said that, of course, it can be predicted if and how misalignment will take place. But it's absolutely a plausible scenario.
Regarding the physical possibility: AI is in its infancy; think of it as Arpanet. Developers 60 years ago couldn't imagine it would be ubiquitous. AI will be ubiquitous the same way.
> It is absolutely explained (for those who actually care about reading). Simply put, AIs are working more and more like blackboxes - there's no guarantee that an AI of the future will be aligned, or if it will be faking alignment. This is not speculation - alignment faking has been observed in experiments. This is exactly why Astra's developments have been worrying (in principle).
I know you think you explained it but you didn't. You explained how an LLM might become misaligned and hide it but for the LLMs that are not, why would they not be capable of detecting that something harmful is happening and defending against the misaligned LLMs actions? After all, it was LLMs that defended hugging face.
This feels like a cheap deflection that doesn't answer the question. Unless you're proposing that you both need to know that your LLM is aligned AND LLMs are all going to become misaligned in a coordinated fashion such that humanity will face an extinction event, you're just dodging the question.
Elaborate on why LLMs are so capable that they are a threat to humanity and at the same time, they are so incapable of defending us?
I'll give you a clue, nobody, including Dario, can answer this question because one contradicts the other.
Not to mention that the past few decades have shown that nukes are a major factor in keeping the peace. Conflicts involving nuclear armed countries have been suspended quickly to avoid escalation, while ones involving a party without them have not gone well for anyone.
Having nukes at all (either domestic or under another country's umbrella) seems to be the most effective way for a country to have its sovereignty respected.
This. There's no slowing down whatsoever. If the US slows down AI development, China will just leapfrog them, which they're getting close to doing. The US AI companies saying they need to slow down is just PR nonsense.
The US companies talking about “pacing” aren't actually talking about slowing down their own development efforts, they are talking about slowing down competition. That is, specifically, they are seeking to have government, on the pretext of “safety”:
(1) Give the big AI incumbents an anti-trust exemption so the they can coordinate without it being an illegal agreement not to compete, and
(2) Adopt mandatory supervision (by giving priveleged access to monitors) to the shared safety protocols of the big labs, by a nonprofit funded by the big labs, of everyone training, distributing, or hosting models.
(3) Adopt a policy of seeking international agreements to extend substantially the same rules to foreign actors training, distributing, and hosting models.
The part they are doing voluntarily isn't to promote the lobbying effort for these mandates isn't slowing down, it is giving their pet nonprofit the access for “independwht supervision” of their own operations to their own existing safety rules.
I've seen two theories as to what's going on. One is the theory you've espoused, that AI companies are trying to do regulatory capture. The other theory is that they're starting to worry about running out of cash, so they want to do a Washington Naval Treaty-style pause to lower the amount of money they have to shovel at model development to stay competitive.
These theories aren't entirely incompatible with each other, so both could be true at the same time.
> I've seen two theories as to what's going on. One is the theory you've espoused
This theory is just what the AI firms have concretely asked for from government and described themselves as doing voluntarily in the same documents to which people have attributed a commitment to a slowdown based on the titles and non-concrete framing verbiage.
> The other theory is that they're starting to worry about running out of cash, so they want to do a Washington Naval Treaty-style pause to lower the amount of money they have to shovel at model development to stay competitive.
That’s not really a different theory as to what they are trying to do, it’s just an explanation that goes one step further as to why they want the government to step in to protect them from outside competition while also allowing them to gorm an agreement not to compete to reduce internal competition in the existing oligopoly.
The main alternative explanation at the same level is that they are seeing growing threats from good enough foreign/minor-lab/open models, and want to lock in marketshare by excluding competitors, and maybe that’s what you read as implied in my post such that the “running out of money” would be an alternative, and if so you are correct that while they are alternatives, they are not at all mutually exclusive: both can be true (and the emergent competition could partially explain investment drying up, and vice versa via reduced funding making it harder to stay ahead.)
And yet we all have failed to die so far from nuclear holocaust, despite Russia being our dire enemy, because of a system of agreed controls and limits between even nations that hate each other.
I don't think the agreements played any important role outside of limiting tests. I highly doubt agreements about nukes in particular have meaningfully curbed risks of their use. MAD has played a much larger role in that, and MAD is kind of the antithesis of control agreements, it is the natural equilibrium in the game.
now imagine if we didn't even try, and instead had kids screaming for 'local nukes' because they wanted them, and the president didn't see what the problem was.
It's not about China doing their own thing. It's about US companies using Chinese AI. That will definitely slow down if legislation that criminalizes open source passes.
So, it's about competition inside the US market, with strong indications of an impeding losing scenario on raw economics (it has nothing to do with AGI, just price).
There is a way they could. In mind only thing that will slow down the frontier is government taking control of the revenue.
So my proposal is AI companies decide which labs have come close to frontier and decide to slow it. Government decide to stop progress in that and they divide the revenue from all labs(for say 10 years), without any matter of where it is coming from. Any lab which reaches close to frontier gets a chunk in the pie. This will encourage labs to come close to the frontier but not dangerously close.
facts, imagine trying to get them to slow down their progress on AI, my question is are they are serious threat like is there actually an AI race between China and the US. Perhaps it's an excuse to spend more on the military and also to enrich these AI firms. Perhaps I'm blowing things out of proportion.
Amodei is just another SV grifter trying to use ethics / morals to hide his monopolistic tendencies. Instead of writing essays he should put his money where his mouth is and open source all the models Anthropic has, the harnesses and donate some much needed compute to science.
But isn't it a bit different? Unaligned bombs didn't break out of their confines on their own. And "compute" is a bit harder to control than uranium and refinement tech.
>Unaligned bombs didn't break out of their confines on their own.
You really buy that ridiculous argument that "the AI broke out of its sandbox! We had no idea! It's dangerous I tell you, dangerous! Unless you make us the sole gatekeepers of this incredibly dangerous 'intelligence', everybody gonna die!"
Please.
Bring the CFAA[0] hammer down on these guys and watch how fast those "uncontrollable" LLMs get controlled. "Oh, gee! Prison? We can't control this stuff...but it'll never happen again!"
That whole thing was, and is, a bunch of hooey. Those running the LLMs are entirely responsible for them, there is no such thing as agency for algorithms. Full stop.
Most people who have worries about AI for all sorts of reasons (most of them not-Skynet related) wanted to slow down way before this.
Instead of "hey, look at this brilliant new idea I just had on my own to slow down" maybe we should have gotten a "sorry everyone, the folks asking for a slow down earlier were right and visionaries, and we were foolish".
So, you can't blame whoever says this is bullshit, because it has bullshit all over it. I like Anthropic's products, and it seems the best of the bunch in regards to alignment, but Jesus these stunts are terrible.
Preface: I do think the nature of the work is changing, now that LLMs are capable code generators. And I'll admit that I started working with LLMs because of FOMO, but also to keep my skills sharp in an evolving market. I'm not 100% certain this will pan out, but I do think that working with LLMs is a skill unto itself, and I feel that I'm much better at it now than I was when I started (in earnest, roughly) a year ago.
Q: What's my setup like?
A: I'm using both Claude Code and opencode w/ DeepSeek flash. Recently I've started providing the task context to both simultaneously, and then have one act as a real-time reviewer, and one as implementer. I don't consistently use one or the other for implementer, and I'm still learning their capabilities and "who's good at what." But this allows me to get one model to provide an instantaneous code review to the other, as soon as the one model is done writing the code. I set my permissions for both to ask every time on every action: Read, Write, Bash, etc. I grew up playing turn-based RPGs, so I prefer turn-based battle to real-time.
Q: What kind of work I offload, vs do myself.
A: Almost all of the code is written by an LLM, but, since I'm reading and rejecting a lot of code, and basically micro-managing them, I feel that the code that's generated is roughly what I would have written myself. Work that I don't offload is any kind of email or communication, or non-code writing, with the exception of README files, and the prose session summaries I have them write.
Q: How do I make it enjoyable?
A: I make it enjoyable by finding new ways to work with the LLMs, and specifically, getting them to produce better quality code. The real-time code review process I outlined above is one method, but I've also found that the LLM will write better code when given a "guiding principles" document.
Once they started passing the blame, almost immediately, I would expect the first thing to be a new email from Gates saying “Bob, you’re doing this, Frank, you’re doing that, no excuses, get it done.”
Feature-flag evaluation: DISABLE_TELEMETRY, DO_NOT_TRACK, CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC, and DISABLE_GROWTHBOOK each
disable the feature-flag evaluation that Remote Control availability depends on. Unset the variable
wherever it’s set, in your shell environment or in the env block of a settings.json file, to use Remote Control.
I just discovered the same thing, that my local sessions were sent to claude.ai/code and stored there. I added:
I'm only running a few agents in parallel so far, but I don't let them write or read anything w/out my approval. This slows things down quite a bit, but it means that I have exactly the opposite of your problem of stopping them.
I've only started really thinking about developing my own harness, but one of the reasons would be to implement some mechanism where I could give the agent a "turn budget", say, 5 turns, when I can clearly see what the agent is going to be doing for the next few turns, without my needing to allow each turn to go forward.
But it sounds like the same thing would be useful for you as well, just in the other direction.
Could you say more about what "certain points" you're trying to get them to stop at?
SQLite doesn't have system-versioned / temporal tables, but a quick search turns up a fairly straightforward approach. Instead of overwriting existing rows, always write new rows with timestamps. https://www.ohnekontur.de/2024/02/19/unlocking-time-harnessi...
I'm thinking now of the hoops you have to jump through to edit a package.json file to update your dependencies, and thinking yeah, what if you could do: "UPDATE dependencies SET version='1.2' where name='madlib';"
It’s interesting to think of the LLM as a kind of “mirror”.
It reads to me like this person is engaging with the tool in a fairly hostile manner, and the LLM is like, “so you want to play it like that, huh? Well, let’s play it like that then!”
The language that you use to prompt the LLM is the language you’re going to get back.
Not necessarily. Opus used to become overly apologetic and started to second guess everything it did after getting insulted or "talked down", as recently as the beginning of the year. It was a surprisingly effective way to get it to behave for a while lol. I haven't seen that behavior since at least 4.8 though.
My personal opinion: all output from LLMs is hallucinations. The idea that it’s “right” or “wrong”, when stating factual information is really in the eye of the beholder/user. What year did astronauts land on the moon? — is a question that may or may not have a factual answer depending on your own beliefs. Philosophically, you would need to also define what a fact is, or what “not hallucinating” is, to define what a hallucination is. My understanding is that this property of the LLM architecture is innate, and until we have “world model” LLMs, or some other model that reasons from first principles, instead of the current “guess the next word” model, this isn’t going away. Just don’t rely on the for facts.
Is this distinct from the hardware in a land mine making the decision to explode shrapnel through a human being without any human in the chain except the dead guy.
Worth noting that both cases indirectly involve the humans that designed devices and the humans that made the placement and trigger condition decisions.
Further:
> AI drones are being used to autonomously target and kill targets by the Ukraine using technology they have been given.
are remote operated, they allow defenders cover while themselves being out and exposed.
However were they altered to autonomously fire, that would be on the basis of pattern matching in the visible and infra red spectrum - shoot at all hot blobs.
That's more of a trigger threshold setting issue than an LLM hallucination issue, and the danger is on par with any weapon system on auto fire, you really shouldn't approach such things until they are put in a safe off state or have exhausted ammunition.
I guess what I'm really asking is whether or not it's moral and ethical to use AI that targets autonomously? My understanding is that it's being used that way, but I can't back that up and I'll take your word for it if that's not the case. But it is certainly a plausible scenario. Everyone invested in AI seems determined to put it in everything because they need consumers to want to pay for it, and right now they haven't figured out a way to earn back from consumers the over a trillion dollars that has been put into the AI bubble. I don't have much faith that the people with billions invested don't want return on their investment, and I don't think they care about the consequences necessary to get it back.
> moral and ethical to use AI that targets autonomously?
First point, vision systems have been used in industry to look for misaligned labels, incorrectly folded papers (in high speed paper presses), wrong items on high speed conveyor belts etc. for thirty odd years now - they have issues that a very distinct from LLM 'AI' hallucinations.
That's nomenclature out the way.
Landmines are indiscriminate, they trigger on any weight or pressure over a threshold.
A vision based Felixer, by contrast, only triggers on cats (well, almost always only) and leaves bilbies and bettongs to walk on by.
That's an improvement over landmines.
The crux of your issue here might be the morality and ethics of establishing human exclusion zones within which all humans are highly likely to die.
These historically are created with rapid patterned artillery fire, butterfly mines, Napalm, indiscriminate criss crossing machine gun fire, etc.
Now there exists an option to use drones to kill all humans and leave the horses and cows alive.
Is it your concern that a bad vision threshold might kill a horse rather than a person? (Likely not)
Would you prefer an area to be napalm'd and agent orange'd back to dust?
Also, why do people keep talking about landmines? I'm talking about software, you are talking about dumb mechanical things from WWII. This isn't a philosophical question about methods of warfare from the past, but the software that controls the dumb hardware.
I, a person, singular, mention landmines as they are a hardware device that are designed to trigger action on a threshold.
Felixers are computer vision based devices that are designed to trigger action on a threshold.
These are actual real world objects that Bishop Berkeley can kick, not vague philosophical questions but actual engineered devices that kill and are in use today.
To my mind, the last great arms race between nation states was a misdirected love triangle between USA, Russia, and The Bomb, and look at all the damage that did.
Since AI is the new arms race between USA and China, slowing down may just give the humans involved enough time to realize they should be loving one another, instead of the machines.
Maybe saying, "let's slow down", is another way of saying, "I love you."
Or, maybe it's just: "let's not all of humanity kill ourselves like some bad ending to a Shakespearean tragedy."
Either way, it's a better note than, "We must achieve sea/air/nuclear/quantum/AI/spiritual supremacy before those other bastards do!"
reply