To be fair, I wish the whole world was much "worse" at warfare in the same way as they are. We'd probably be a much happier world overall. Or at the very least less destructive.
I tend to agree that when used correctly, a surveillance system like this could have benefits at a societal level.
The problem is the many stories we already have of misuses of this equipment, showing that people in power are not mentally capable of actually managing a surveillance system like this in a societally beneficial way.
If the use of the system was designated as only being allowed to stop heinous crimes and we had a track record and way to prove this was always the case, then I would be 100% in support. But given it has already proven to not be possible is likely why most people are against it.
I simply don't think Humans are capable of proper use of a system like this.
The problem is that these system are setup specifically to be able to be used incorrectly. Flock is getting all the rage right now but they aren't the only vendor. An acquisition and rename later and everyone will have forgotten this all. Police (others?) are setting up stingrays. Stalin's secret police had a maxim: "Show me the man, and I'll find you the crime." Remember, everyone who installed a Ring doorbell camera paid to help set up these kinds of networks. (all so that a porch pirate might get caught even though Amazon will reship for free). These systems also set up a huge target for bad actors to hack outside of the law.
If these things were only ever used with the a valid search warrant, that would be a great step forward. But even that will only stop "honest" people from abusing the system.
> Audio used for wake-word detection is processed locally on the TV and, if no wake word is detected, audio is not converted to text, stored, or transmitted. Voice-recognition results and related technical logs may be generated as part of processing a voice command. These records are associated with specific voice interactions and do not indicate continuous recording of conversations occurring outside an active voice recognition session.
From their statement. This entire thing is contradictory. You cannot both say "not converted to text, stored, or transmitted", while at the same time saying the results may be generated and stored in the form of logs.
We need to hold companies responsible for this shit.
If someone is genuinely afraid of this, they wouldn't IPO in the first place. All the talk in the article about commercial incentives means nothing when the company plans to IPO and become beholden to investors.
Aren't they beholden to investors already? Why would an IPO make a big difference?
In any case, if they need to IPO in order to survive as a company, then bringing in regulators to act as a counterweight against commercial pressures makes a lot of sense to me.
If anything, his entire argument leads to the eventual premise that AI labs need to nationalize.
He is already arguing that commercial competition creates dangerous incentives, and that labs cannot slow down because competitors may overtake them. He also asks for what would boil down to significant government intervention to preserve a Western lead.
If this is true, why preserve the commercial incentive? The U.S. government would already have the necessary goal of remaining ahead of China and other competitors. Why not remove entirely domestic race for market share, valuation, and investor returns rather than keeping private labs in competition and then asking regulators to counterbalance the incentives that competition creates.
That doesn't mean nationalization would be better, but given the severity of the risks he describes to national security, it seems like an obvious conclusion that nationalization is the eventual end result of the argument.
I don't agree that the argument implies nationalization. Regulation would work if it slowed everyone down at the same rate, enough to mitigate the risks. The problem is that you need regulators who know what they're doing and strong international cooperation. If you regulate a fraction of the global market, you just create an incentive to shift development to other countries. Nationalization, being essentially the most heavy-handed form of regulation, faces the same problems and comes with its own risks as well.
We're talking about this like all of the bad incentives are created by market competition, but that's not really the case. Most of the incentives come from untapped value in the form of potential profits, strategic advantage, military superiority, etc. Corporations and governments want to capture this value for themselves, creating various types of competition. Dario's argument depends on the assumption that the primary risk comes from the pace of development and threats from the technology itself. That's probably where I disagree the most; I think the highest risk is rising authoritarianism and competition between nation states. Slowing down isn't really a solution to those problems.
I think this is the correct take. And until we have an AI Hiroshima it’ll be difficult to get the international community to work together. It’ll take rogue AI (or AI-powered group) disabling a significant world power before everyone comes to the table. Otherwise, it just looks like MAD and the equilibrium holding to the powers that be.
It wasn’t the shock of Little Boy that ended the war. That was just a final chapter of a 10+ year journey of horror, despair, and destruction, and the impact of Little Boy cannot be considered outside of that journey.
Europe had been destroyed by June 1945, and yet the empire of Japan continued to fight.
If that pattern holds, a single malicious AI substantially disabling a single world power won’t end the AI race. Participants don’t learn by the defeats of others.
If you’d like to apply a metaphor maybe the 10+ years of world war is more appropriate, after which basically every participant save one was exhausted.
Anthropic is structured as a Public Benefit Corporation with a strong charter. In addition, founders will have super-voting shares and so it won't be possible to push them out. Therefore the whole "beholden to investors" thing is not a concern.
Don't underestimate the ability of the courts and the state to pull the rug out from under the legal system. Speaking as an outsider from Canada, it looks very much like all bets are off in terms of respecting checks and balances in the United States and it's very realistically possible that unless there is a big upset and turn around the next couple of years any traces of democracy in the US will be a farcical nod to what the founders built as a way to paper over the abuses.
I really hope I am wrong as my perspective is not anti-American, it's specifically anti-corruption and pro-democracy.
As long you are burning more cash than you bring in, you are beholden to investors - whether it is retail, VCs, banks or a government giving you a bailout it is still someone signing you a check.
Golden shares, vetos, PBC, charter are all paper tigers , they only matter if/when the firm is self-sustaining business with no outside capital needed, the alternative to not listening to investors till then is crash and burn.
After that point, you will have to listen to the paying customers (sometimes but not always they are also users ) as they are ones now funding your organization.
Assuming they can achieve funding without IPO probably yes. But if they can't there's always the dilemma that "if [good guys] won't do it then [bad guys] will". I'd like to think that the leadership at anthropic is principled even if the actions of the company as a whole has been less than stellar morally speaking. I'd be curious to see their moral calculus transparently laid out in public.
We are still living in a capitalistic society. We have to find a solution which is responsible, safe and returns on the investment. The commercial aspect can be true at the same time.
The potential of AI may give us the opportunity to graduate to a post-capitalism, or to move to a sort of neo-capitalism, which is governed by different rules.
Capitalism thrives in the realm where there is a scarcity of resources, either physical or informational. Imagine breakthroughs in energy science such that the cost of energy drops to zero, which means the cost of physical resources declines precipitously. Ok so where is the capital now? Must we retain a model based on the premise of scarce physical capital?
you are imagining magic as a reason to dump all your money and future money into a casino. You might instead consider joining the catholic church? they already have a free energy god that gives everything you could ever want
the cost of energy is already ~0 and people dont want it, and refuse to participate in letting other people have free energy if it affects their view of their pasture.
unless you are building killer robits with the intention of doing some soviet or nazi styled purges of everyone that might get in the way, you arent gonna get this ai utopia
Read other writing by Anthropic about potential future financial implications of AI. [1]
IPO is a vehicle for future wealth distribution. It leverages existing financial frameworks in order to span the gap from before-to-after without having to convince the system that future is inevitable.
I don't think Anthropic and OpenAI will IPO at all. Word is that Anthropic will have $100 BN in revenues this year, and very likely OpenAI will get some similar amount. You IPO when you need money, and I think Anthropic and OpenAI are past that point.
> Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.
It's almost like the people at Anthropic and other AI labs are slaves to a misaligned reward function that encourages optimization of a metric (profit) over human values, even at what they claim is the risk of extinction.
We built the paperclip maximizer, and it is capitalism.
> are slaves to a misaligned reward function that encourages optimization of a metric (profit) over human values, even at what they claim is the risk of extinction
This was already the case even before the "frontier AI" age. The big question is, will these glaring AI arms-race threats be obvious to enough people to rethink the underlying systemic flaw driving it all?
> Crack down on unauthorized distillation by companies in authoritarian countries. Distillation of frontier models allows lagging companies to narrow the gap using a fraction of the cost it would take to develop their own AI independently.
This and related quotes seem to attempt to place some of the burden on the US Government as opposed to themselves. It seems to be a trend that Anthropic wants to be able place some of the burden of preventing distillation on parties other than themselves.
Preventing distillation is fundamentally their problem. Whenever it happens, it is primarily due to their security efforts not being able to prevent it. I'm tired of seeing them play the blame game and divert responsibility. Sure there is a level of national concern, but the response could simply be the government placing stronger controls on Anthropic themselves. If they are unable to prevent distillation, then the solution may be to literally limit their distribution until they are capable of doing so effectively.
> A race to the bottom, spurred by commercial incentives, can make these risks more acute
I also think it is ironic that they're planning an IPO while also talking about a race to the bottom due to commercial incentives. These are fundamentally at odds. Being a public company means you are beholden to investors and a board with the primary goal of making more money. What higher commercial incentive is there than that.
If commercial incentives are as dangerous as he states, potentially becoming one of the largest public IPO's and largest traded companies in history a pretty strange way to reduce the risk of commercial incentive risks.
This is also one of those scenarios similar to subscription or membership cancellation where we should not allow accepting/signing up for something require using an entirely different method of communication to cancel/opt out.
Requiring written notification while the TOS being accepted entirely via digital means should be illegal.
Not to mention I think its insane that we allow companies to be able to dictate how citizens are allowed to utilize their legal system.
Your access to a digital product you have "purchased" should not be tied to the continued existence of the store you also bought it from. Similar to physical products.
If I bought a game at GameStop we aren't required to continuously reach back out to them every once in a while to check that they're okay with me playing the game I bought from them. And if GameStop goes out of business they aren't going to come knocking on my door to make sure I never play the game again or take it back from me.
Once a game or any other product is purchased from a digital store, it should not require the stores continued existence or permission.
Yes there are some drawbacks in that maybe they could argue then that they could stop allowing redownloading a game you've already purchased but so be it, they'd just be offering a worse experience and users would move away.
We also need to stop allowing unclear language. When we lease a car at a dealership they never use the term "buy" or "own". That's strictly reserved for ownership. We should require the same for other licensed/leased products.
One fun thing I remember from reading some EULAs for physical games I bought in the past: the EULA for StarCraft at least explicitly said that you are required to obtain prior written permission from Blizzard Entertainment before lending the disk to a friend. Obviously this is not practically enforceable, but it shows that nothing of this kind is new from these companies' perspectives.
The way the law is structured, you are never buying copies of digital goods, regardless of the format in which you obtain them. Instead, you buy a license to use the digital good under the specific terms and conditions of that license. That is why if you get a disk with GNU utils on it from the FSF, you are still bound by the GPL in what you can do with the data on that disk.
It's ridiculous to me that trying to play some recent games from Microsoft-affiliated studios requires me to sign in with a Microsoft account even just to play also malone. Paying them for Halo MCC or Gears 5 isn't enough to be able to play the campaign alone; the main menu is literally blocked by a sign-in screen. There's no reason whatsoever this should be allowed.
> In the digital age, it is not plausible to allege that reasonable consumers believed they were obtaining "ownership" of a digital game.
So their argument really is that it is unreasonable for anybody to believe they own any of the things they download or purchase digitally? What a leap there.
I mean thats as plain english as you can get to an official rubber stamp on "you will own nothing and you will be happy".
And then they want to have it both ways by interchangeably using the words "revocable license" and "ownership" when it suits their needs. License in small writing/text, but "own" in anything the user might actually reasonably read.
Plenty of software/tools that we use are as good as they are despite being run by entitled pricks.
Some would even argue that in certain cases (apple being the biggest example I can think of from the past at least with Jobs) they are as good as they are almost due to the megalomaniacal people who run them.
reply