I’d roll my eyes if the engineers stated that they didn’t design the reactor to melt down, and that it simply developed rogue meltdown-desiring behavior on its own, and I would also wonder about negligence if they claimed that nobody could have anticipated this (given that, like with botnets and viruses, we have decades of knowledge and experience regarding reactor meltdowns)
I mean sure, negligence is absolutely on the table; but that makes the problem worse, not better! We don’t allow nuclear engineers to be negligent; they can go to jail if they don’t follow strict protocols to make sure the dangerous systems they work on are safe.
I suspect that if you're not careful with agent memory it creates a danger of agent-driven cargo-cult behavior. I've watched this in my own ad-hoc agent loops where it starts with something basic, like the first agent tried to run some gigantic dependency inspection command and OOMed the local JVM and eventually recorded a workaround (to enable it to run gigantic dependency inspection commands...), and by time I get a few more agents into the loop, agents have written entire paragraphs about testing and validating local dev environment memory configurations that are mostly irrelevant to whatever is being worked on.
In general I've seen other issues like this where small errors and irrelevant comments in the codebase spin out into larger problems that consume annoying amounts of time/tokens. Maybe Anthropic and OpenAI don't notice this because they're in an "infinite monkeys with typewriters" scenario, but it's noticeable to me when the agent in my CLI has been spinning for 15 minutes contemplating irrelevant details
This happens way too often in my experience and I tried a lot of things. If the agent has access to an existing codebase it will always try to stay as close to it as possible. It doesn’t diverge from the initial direction the project/code went into, even if it’s already clear that this direction goes nowhere. Somehow agents aren’t able to detach themselves from existing code to come up with completely new solutions.
Sometimes I feel like it would be best to only give the agent access to clear API boundaries (say public interfaces to certain modules) and let it work out a new system from scratch given the expected inputs and outputs. Then plug this independent solution into the system. Of course it can still overfit these interfaces but it’s less than having access to the entire codebase.
Thinking about it, maybe it’s possible to let one agent extract those interfaces, then use them as grounding for a new session.
One less nice way of achieving the same is to tell the agent once in a while that it should think completely from scratch (from first principles). But this relies heavily on instruction following in the reasoning part, which sometimes works and sometimes fails.
With Claude it feels like those engagement invitations have turned into things like "By the way, it's worth sitting with this problem I've identified in your assertions..." or "Here's an alarming gap worth resolving in your code..."
Gemini also does the same lighthearted GPT-style invitation with the default prompt in the Google webui, but it doesn't seem to exist on the API. The Claude models seem to have been trained to force this structure on every one of their responses, and until I realized they always stick the same thing in the last part of their response, I found the Claude version more distracting since it's always pointing out an imaginary and supposedly very important problem.
The site collection seems pretty random. There's a mix of actual AI labs, extremely questionable resellers (like whatever "claude-opus.top" is), and then random consumer sites like baidu and xiaohongshu.
In addition, many Chinese companies are trying to give their programmers access to Anthropic models even though they're legally prohibited from doing so. And that might involve employees using unmodified Claude Code with an ANTHROPIC_BASE_URL pointing to a proxy on the company intranet. In Alibaba's case, I've been told by an employee that they went the extra mile of setting up a hermetic cloud environment where employees could indirectly use Claude Code without ever having it touch their work computers.
Anthropic does their best with banning accounts. As the result, shady API reselling market emerges. OpenAI on the other hand doesn't really discriminate based on a country like that (but a VPN is required nevertheless).
GGP said "legally prohibited" not "against terms of service"
Keep in mind the only law that applies to them is Chinese law, so even if violating a term of service was illegal in America (it isn't) it would also have to be illegal in China to justify the statement.
It’s interesting that with both Anthropic and Google we’re seeing them develop agentic models that are supposed to do anything a human can do on computers without human intervention, but at the same time, if you plug one program into another of their programs or APIs in a way that wasn’t preapproved you may be blocked or banned.
To be charitable, maybe they’re expecting AI agents to eventually start reading the ToS docs
I wonder if you can use lower quality models (or some other non-llm related process) to inject more "noise" into the text in between stages. Of course it wouldn't help retain uniqueness from the original source text, just add more in between.
The vibes around the self-driving car hype (maybe 10 years ago?) felt very similar to me, but on a smaller scale. There was a lot of "You might like driving your car and having a steering wheel, but if you do, you're a luddite who will soon be forced to ride about in our featureless rented robot pods" type of statements, or that one AI scientist who was quoted saying we should just change laws around how humans are allowed to interact with streets to protect the self-driving cars.
Not all of it was like that, I think oddly enough it was Tesla or just Elon Musk claimng you'd soon be able to take a nap in your car on your morning commute through some sort of Jetsons tube or that you could let your car earn money on the side while you weren't using it, which might actually be appealing to the average person. But a lot of it felt like self-driving car companies wanted you to feel like they just wanted to disrupt your life and take your things away.
I've triggered similar conversation level safety blocks on a personal Claude account by using an instance of Deepseek to feed in Claude output and then create instructions that would be copied back over to Claude (there wasn't any real utility to this, it was just an experiment). Which sounds kind of similar to this. I couldn't understand what the heuristic was trying to guard against, but I think it's related to concerns about prompt injections and users impersonating Claude responses. I'm also surprised the same safeguards would exist in either the API or coding subscription.
I look at products like Hershey's chocolate or Reeses more like their own category of processed food, kind of like Spam. They have a close, but not exact resemblance to "normal" chocolate or peanut butter, but they're also sort of an acquired taste, and I think their customers would be upset if Reese's Peanut Butter cups suddenly tasted like the Trader Joe's versions (with real peanut butter instead of a mysterious chalky peanut-flavored substance), or if Hershey's stopped using the butyric acid process that makes them taste like vomit to non-americans.
reply