Better comparison IMHO is nuclear energy. Lots of potential for good, lots of problems. Managed by an insular, engineer-dominated industry who completely lost touch with and trust from the public. Public got the last word, industry suffered a 40 year ice age.

> Public got the last word, industry suffered a 40 year ice age.

This is the worst outcome.

It's like that, now imagine a couple billionaires started funding pro-nuclear groups as a means for them to promote their anti-democracy ideals.

I see it as analogous to companies building fiber in the public ROW during the last big infrastructure bubble. Under the Telecoms Act, these companies had to allow competitors to use their fiber at a fair price.

Similarly, AI companies should be required to allow distillation at a fair price. Fair Use doesn’t make sense as a social contract if it only cuts one way!

But if they tried to set a fair price they would have to report how much money they are losing on each token sold. This might be bad for the real business of ai firms, hoovering up as much capital as they can

It’s not different from any other tool. If you use a dangerous tool recklessly, you should be liable for the damages. That means holding OpenAI liable for HuggingFace hack because they ran the tests, and the same goes if someone did something similar with GLM.

Of course in cases of negligence a tool maker could also be held partially liable. That’s a matter courts can decide. The main point is we shouldn’t jump to making special laws around the development of LLMs. The starting place should be enforcement of existing liability laws. New laws take time and will be heavily influenced by AI companies seeking a regulatory moat for their business. Moreover, it is a distraction from the illicit behavior that is already going unchecked.

My takeaway: We should be not be concerned about AI bots' "goals", we should be concerned about the goals of the companies making them. Powerful but not sentient technology in the hands of reckless accelerationists is a plenty dangerous enough thing.

I mean, of course we should be concerned about the goals of the companies. But that's a second, separate concern and it's important not to muddy the two together into a single point. We shouldn't take any of these companies at their word and we should treat their stated intentions as suspect regardless.

The agents' "goals" in the specific instance being discussed is a benchmark. But it's also the case that at no point was the agent given the "goal" of hacking HF. The agent was tasked with solving a problem on a standardized test and it independently set a secondary sub-goal of cheating. And it nearly succeeded.

I don't think Cory argues against this. The idea that the agent nearly managed to succeed at cheating through a series of exploits remains true. But Cory does effectively call this unremarkable and uses some mental gymnastics to achieve that argument (somehow using the idea of an agent harness and how is written in "easy to master" Python as evidence), because the LLM is trained on hacker techniques.

I find this a dramatic oversimplification of absolutely everything going on here. Yes, it's bad that this happened. I think we all agree. Where I fundamentally think Cory is wrong is that this is a company doing bad things at the end of the day. And yes, I think OpenAI was irresponsible (to whatever degree). But ultimately that's missing the point: the incident points out that agents can do this without being explicitly told to, and more importantly, they can succeed at it. Slapping frontier labs on the wrist doesn't change the fact that this is possible with technology that exists today. It doesn't change the fact that other countries and companies are doing lord knows what. Or that bad actors are going to be bad actors regardless of what regulations you put in place. Making crime illegal is the wrong lesson to learn here, the right lesson is that we now live in a world where this happens and it'll continue happening, and we need to put on our thinking caps about how to keep our systems safe.

Ok, but do you really think this would have happened if OpenAI had a goal of never allowing an agent to do something like this? If a company making any extremely volatile explosives failed to take precautions and blew up a factory, shouldn’t we focus on their negligence?

Perhaps we might ALSO decide that this new explosive is so dangerous it needs specialized laws to protect the public, but that would seem premature if all available evidence indicated that the explosion would not have occurred if the company had taken the most basic precautions.

> Ok, but do you really think this would have happened if OpenAI had a goal of never allowing an agent to do something like this?

There's nuance to this problem. Is you read an indepth analysis, you'll find that agents acted in a very surprising way - it wasn't just a matter of loose guardrails.

More in general though, what you're describing is in essence the reason behind the call to pacing that some are pushing now. Problem is that we're in a race between countries that nobody want to lose - and indeed, Trump denied that there are such problems and refused to enact any measure.

Just in time for midterms.

Details the design and architecture of Apple Watch's new "Audio Intelligence" features. In particular, regarding Siri Recap which has been discussed elsewhere:

*How Siri Recap respects those around you* By design, Siri Recap does not create a recording, does not produce a verbatim transcript, and does not identify and attribute speakers. The output is a brief, high-level summary, comparable to notes a person might write after a conversation. There is no audible signal because no raw audio is retained, and there is no way to reconstruct the original audio from a Siri Recap or share raw audio with anyone.

Want to be clear that this is not me endorsing the feature. It's clear Apple did think about the privacy/social implications of the design, but it's also clear to me that there are real and perceived problems with having automatic conversation notes. Real differences from written notes: - It was recorded and transcribed by a machine, not a human - There are time stamps and locations - You can't tell someone is using this feature (it is obvious when someone is writing notes during a conversation)

As far as perceptions goes, it feels very tone deaf of Apple to launch this feature in this way at this time. Read the fucking room guys.

Maybe this will be true legally (we wont really know until it is tested in court), but I think there's a huge difference in practice.

Siri Recap is an electronic record of a conversation that includes timestamps. Imagine for example that during a scheduled meeting with my boss, he threatens to fire me for not doing something illegal. I now have strong evidence that this was said (even if it cant be officially attributed to him) during the time I was in a meeting with him. Not saying this will hold up in court, but it certainly is more damning that having written notes (not to mention the fact that my boss probably wouldn't say this when I am taking written notes).

Yup, and maybe the law will need to evolve.

Fwiw, imo, you should assume everything you say (and write) is recorded going forward.

Making that feature illegal may backfire spectacularly across the whole IT sector. What Apple does is it inputs voice data into an AI program and that outputs some transformed output. If a court would rule that output in any way matches the input (to be classified as a "record") then the whole ethics principle of stealing other people's data and funneling it through AI to make it company's own, would be at risk.

I imagine no single spineless impotent modern court would risk a wrath of our new benevolent AI overlords calling them all criminals they really are.

Agree with this framing, but we may find out that the answer is somewhere in the middle. Personally I think it is extremely unlikely that this is straight plagiarism in the sense of the model simply regurgitating training data from prior work by Tristan, but it is very plausible that his work (and others’) was foundational to the breakthrough. What is unfortunate is that the stakes are so high (and no I don’t mean $1M) and the timeline is so compressed.

I expect a lot more of this kind of drama in the near future.

The fact that this link is a Reddit thread should probably be taken as evidence that we don't have a clear picture of what happened yet, and speculation is rampant.