the sad thing is we need AI for shopping because it has been in the interests of online sellers to make a hostile experience on purpose - when I go to Amazon and search, the first organic result is almost scrolled off the screen past all the sponsored ones, or when want to buy from a random seller I'm almost definitely getting shunted through an account signup I didn't want or fooled into an affiliate purchase some other hostile experience beyond just "buying the thing".
So now we have AI to overcome the hostile sellers, but the fact the sellers introduced the friction in the first place strongly suggests it will just come back again in some other form. It wasn't there by accident, it was serving people's interests and once AI vendors have finished getting consumers hooked in, they will then turn around and enshittify by giving the sellers back some of the friction - for a cut. So you won't be able to just order what you want without the "would you like fries with that?" or "what about this other brand?" coming back.
> the fact the sellers introduced the friction in the first place strongly suggests it will just come back again in some other form
Guarantee they'll introduce some dynamic pricing shenanigans where they charge more when they identify that the shopper is an AI agent. Unsupervised AI with access to your credit card is basically the perfect target for price gouging.
I only really used it because the harness was way too trigger happy to start making changes. Even if I just asked a question some times I would come back and it refactored the whole codebase. Now it doesn't seem to do that any more.
I still would appreciate a "read-only" mode. It's not uncommon that I start a harness ONLY to explore and understand the code and I don't really want one typo to have it off building something, or even to save a plan document.
It's pretty clear something was left unsecured and the agent just "found" it
This is going to be something long the lines of someone coming in to your house after you left the door wide open. They should probably not have done that, any respectful person would not - but calling it a "breach" is really too much.
The title is misleading - it sounds like they specifically went there to harass employees by filming them, and Meta took down the videos for harassment, like they routinely do if this is reported on their platforms. I'm really not sure what the issue is.
Meta only cares about this video because it directly targets Meta. If this guy was asking questions to employees of any other company they would not give a shit.
I have read news article about people wearing meta glasses, removing the red recording LED and filming woman and girls at beaches in bikini without their knowledge. And these video often stay up on metas Instagram, accounts don't get suspended. THAT is the issue.
it's really weird to me at the moment because both OpenAI and Anthropic seem to be competing in an extreme benchmaxxing contest on super intelligence that actually nobody cares about. I haven't really cared about model intelligence since about Opus 4.8. It is by far not my biggest problem. I don't need to replace or support Einstein in my production workflow. I just need basic intelligence that can equal a routine office worker - safely and reliably. What they doing - chasing super-intelligence but dramatically escalating risk - is actively what I don't need.
I really think they have drunk too much of their own kool aid and become completely detached from what the market wants.
Affordability is derivative of control which is really what I care about.
I'm just not going to build long term infra that depends on something that another person can and will - objectively based on experience - take away from me at some unknown point in the future.
The biggest benefit of open models is they keep all the other players honest. The extent to which they feel they can dictate terms is directly set by the threshold where they feel people will take the trade to run open models instead.
It's funny, i push back on pull requests because there is too much description now - a 20 line change has pages and pages of generated description, rationalisation for why it is safe, defense of each design decision, analysis of risks and side effects. People are indignant, you're rejecting my change because there is too much documentation? And my response is, I don't have time to read it and you put me in the position where I can't afford not to - because approving the PR implies I did and accepted it. The investment to read all that for the value of a code change that I'm one prompt away from doing myself if I cared is just not high enough. So it's rejected.
You might be surprised how many organisations/ teams now don’t even review pull requests. Claude does coding, Codex do code review, and if developer did 5 round of this before pull request, we could very well just let CI merge.
I'm very curious how this goes long term. I guess we will find out.
My instinct says that these systems will expand their complexity to fully fit the cognitive budget of the agents that coded them and then atrophy the same way human-built systems do at lower cognitive budget. Only this time, because of the larger up front budget, the complexity ceiling will be higher, and the potential depth of the problem may be much much larger. It may mostly manifest as increasing cost over time - the agents grind for longer and longer, iterating over and over to fix all the failing tests, and the breaking point will be where it never converges and you come back to millions of dollars in budget spent and still tests are failing and effective gridlock on system changes.
But this may be all my human-biased fantasy that justifies still taking a role in software development.
> ... these systems will expand their complexity to fully fit the cognitive budget of the agents that coded them and then atrophy the same way human-built systems do at lower cognitive budget. Only this time, because of the larger up front budget, the complexity ceiling will be higher ...
That is exactly how one could describe the relationship between heroin and the users health systems. It’s okay for a little while, until it isn’t but there is no going back. When the risks start telling, lawyers will have a field day. Assisted by LLM’s obviously, which then puts the burden on judges, who then need LLM assistance.
This thing called attention economics has captured my attention quite hard. Digital opiates are everywhere since 10 to 20 years. And in my view, the part that they implicitly of explicitly strive to capture your attention is new.
Well articulated. I tend to think that the outcome will vary hugely by project. We already observe this with some reporting certain models behave in a certain way, whilst others see the opposite.
To use the wooley term “quality”, the top 20% might stand a good chance of making huge strides. But the remaining 80% of projects (in particular the bottom 20%) will atrophy extremely quickly. Yet, these will be the project that many push LLM’s too as their domain/technology is complex and/or outdated. Digital transformations that can be done quickly will be tantalising but ultimately unsatisfactory long term (as you describe).
IMO, PRs were never the right process to use in tightly-collaborating teams such as most companies. PRs got popular because orgs started using GitHub, and GitHub had made that workflow to fit the needs of open source (and modeled it after what open source was already doing in the days when patches got sent around via mailing lists).
I'm old enough to remember using CVS and then subversion in companies. People would commit straight to main (which was then called "trunk"), because making feature branches and merging them was cumbersome. And, on regular intervals, the person responsible for some corner of the codebase would do a show-and-tell presenting it to peers, but without the sharply defined boundaries of what the code looked like before vs. after some recent set of changes. People might remember some things from the previous show and tell or from first hand experience with that code, but that kind of memory is necessarily fuzzy, and diffs weren't an artefact that was typical to look at. So, these reviews didn't block people, and any comments that came from reviews defined a direction that things should go from here on out. If a corner of the codebase was deemed to be in a bad shape, the blame around that was equally fuzzy.
Others are just implicitly doing this but pretending review still exists.
Everyone is fatigued by endless code review which you get no credit for and has become massively more of a burden.
All PRs are superficially fine now. There are no typos, there is unit test coverage, but there are deeper issues that require massive amounts of effort and time to spot.
Unfortunately, a lot of previous PR reviews already were just gatekeeping, or "presenteeism". People would leave comments about class names or method names, like they couldn't understand what an `apply` method, the only method, meant on a class with that was already named appropriately and did one thing only (to give an oop example). No, the method needed to be renamed `PriceChecks.applyPriceChecksWithTimeConstraints`.
Lots of review comments about various conditions that wouldn't feasibly happen (same shit with claude now).
But then I'd see these same reviewers approving PRs where the bigger design was just fundamentally broken. Oh, we're adding a blocking call on our hot path, but at least the method name makes it very clear that it is blocking.
In general I agree that the current AI reviews are creating too much noise and it is masking these bigger design issues.
and this is such a liability imo. I recently got fired for what I think was apparently merging a change to Kubeconfig that both the CTO and DevOps person approved. I made as damn sure as I could that the proposed change was good, but I needed their eyes on it and they clearly weren't, possibly be cause every other repo had a review bot and no special automatic deployment automations. The DevOps person literally commented and said "It's good to merge", who to trust
I also have ADHD-driven blah blah blah. I find achieving brevity laborious, but find LLM results to be unacceptable. A journalism class taught by a well-known professional food critic helped most, but consistently using the free “Hemingway App” editor was almost as beneficial. Blindly following every suggestion pares your writing down to a nub, but it’s great at highlighting sentences you need to rethink without blurring your voice and ideas with bland suggestions.
That's a solid recommendation, thanks. I also did journalism. So I can write succinctly, I just have to put effort in like you. Which is much more than I'm willing to put in after I just wrote a treatise on the human condition to send to the lead developer at 07:30 on Monday.
Lately on my hobby projects I've been lazily committing work with the prompt 'add the relevant files to git and commit with a message that explains why' - only because I remember 15 years ago reading an HN comment from someone complaining that too many commit messages answer 'what' but not 'why' haha
So far I haven't had a reason to go back through commits to isolate any issues but if I do hoping the 'why' messages may come in handy for my LLM lol
I just have my LLM read the PR, then give me a summary of what the PR is about, how important it is, and how good the PR itself actually is. Then I have a conversation with the LLM about specific points, especially things where I get that feeling that I don't have a 100% understanding.
Until I fully understand what's going on, the PR doesn't move and my interrogation of the LLM doesn't end. My interaction is littered with "Explain X" and "How does this square with Y?" and "What if Z happens?"
The interrogation is the point, without me having to wade through hundreds of lines of irrelevant code to get at the meat of the matter.
Personally I'm starting to see less and less "smart" people here, given how easily and quickly everyone jumps on random hype trains with very little investigation or thought put into it. And those who do investigate and put thought into it are then quickly down voted for being a hater. This forum has turned into almost like a FOMO echo chamber in a way.
I can read the room. Coding is going to go the way of some other disciplines where machines do most of the detail work and and we know stuff works by verification. There are other fields like this.
Do I like it or not? That doesn't really matter. I need a job, so I'm going to get good at the new way to ensure I continue to have a job. I consider this to be a smart thing to do for myself and my family.
I agree with the work changing more towards verification, but if the verification is one LLM reviewing another one, then that isn't really verification. At best you "assume", but you don't "verify".
Unfortunately I see a lot of (senior as well) engineers who think that just a vanilla LLM reviewing another LLM is sufficient, and my comment was directed towards those. If however you see the LLM era as needing more test support and systems than ever before in the form E2E tests and so forth, where "code review" as such becomes mostly irrelevant as you have such a strong test system in place that if that passes you can be sure it doesn't break anything for users, then yes, that's good.
What I mean is verification through testing, and perhaps other more formal means. This has happened to other engineering domains, yet us programmers have so much hubris that we think it can't happen to us. And everyone is throwing their toys out the cot because it looks like it is actually going to happen.
Code reviews may not even happen, or if they do, it'll be all automated, and the verification will be the key.
Ask anyone in the semiconductor industry when last they understood the design of those things.
At this point, what is a human dev even there for?
We have this at work : fully AI-generated code and description. People will give review comments generated by AI which the "author" replies with an AI-generated response, all with LLM wording full of jargons no one understands not even the person who sent it. When you ask them what they meant, yeah idk Claude said so
The human is there as the control valve, the one who keeps the overall context, and the one who injects actual creativity.
Every time the LLM throws jargon around, you call it. "What do you mean by gated wedge?" You call its bullshit, check what it's saying against your understanding of the overall system, and keep it on the straight and narrow.
It's a lot like supervising a junior dev who happens to be very quick at absorbing lots of info, but not so great at the big picture.
I’ve started just asking it to touch up text to make it sound more human and follow technical writing guidelines and it converges on convincing pretty quickly. It also makes it easier for me to verify that it’s correct because all of the stupid LLM noise and formatting goes away and it gets distilled down to the important bits.
Commit messages as you describe ("fixes", "updates") are inappropriate in any professional context and some coaching should occur to the people doing them.
I have the opposite issue - some of my team members now submit mini-essays generated by the LLM. Like 300-500 word commit messages with everything from the essence of the change up to philosophical design trade off discussions.
Like most writing, what is left out is as important as what is included.
> we assume there is a correct compute graph (a “workflow” represented in code) and use the predictions of the largest, smartest, and most expensive external models as reference probabilities.
...
Rephrased: every model gets the same workflow. We test how they compare to the average of the smartest models (in this case, Astra and Fable).
They assume there is a correct graph, but they don't compare to that, they compare to the average of the smarts models? So the smartest models are getting it wrong but you compare that anyway as a benchmark? So the outcome is "how much of a Fable am I getting" etc. Why not compare the actually correct thing?
But then even on this hand constructed eval, the first plot is showing Jev at less than Sonnet 5 accuracy. It is barely better than Luna. There are two Opus 5's and two Sonnet 5's without explanation. What is the plot showing?
It would all be more convincing if the incidents so far didn't seem to be facilitated by an outrageous level of negligence.
We had OpenAI "accidentally" run an entire swarm of 10,000 agents apparently for weeks, on a security related task, seemingly totally unsupervised, hacking all over the internet - all the conversations were completely visible, anybody who looked would have seen it. But they didn't.
So before we start regulating innocent parties, maybe let's start by taking some direct action against the specific ones that appear to be behaving with criminal levels of negligence.
The "sandbox" they used was apparently made of thin paper exposed under a day of heavy rain, too. You'd think, if they truly believed the model is so dangerous, they'd run it in a VM without a network adapter.
You don't need a full air gap. Set up a microVM with network access limited to local network and send all package requests through a filtering gateway that only allows normal download endpoints. Or self host a big collection of popular packages if you need extra security.
No that's not what they did, they exposed jfrog raw. It would have been so extremely simple to gate services they need the llm to access... I mean, jfrog was not written with this kind of threat model in mind, and neither were a lot of other tools
Right, you mean it didn't go through a gateway? But would that actually have helped? The requests all went through jfrog didn't they? I guess it depends on the level of filtering at the gateway?
Whilst it might not be JFrog's threat model, I wouldn't assume it can be used as a full internet proxy.
I don't really mean to defend OpenAI here, but they did make some attempts at sandboxing. Although it does seem that they didn't really know what they were doing.
It would have been a case of isolating exactly what functionality is needed and wiring that up with the actual requests. Not like a full pass-through proxy. This is what we've been doing in our company as well
> jfrog was not written with this kind of threat model in mind, and neither were a lot of other tools
And we don't just magically know all the consequences of that.
Which is exactly why we do need full, physical air gapping. (Which, yes, would also include self-hosting a mirror of the package repo, if the point of the simulation is to see what's possible with the real package repo.)
> Agents need packages like the rest of us. Ruby gems, npm packages, Maven, pip, docker images..
Yes, yes they do, but read through artifact proxies are dodgy as fuck, which is why and facebook (and I assume a fuckload others) don't have them.
Also semi-airgapped labs are a lot less expensive than you think at that scale. Once you have to do multi-region VLANs with machine certs before you get access to juicy VLANs, the difference between "no internet for you" and "mostly airgapped" falls to almost zero.
Also I would want an artifact mirror because a) that give a good signal about how the model reacts, and what training material its latched onto, b) it hides what the models are doing from the outside.
The best part is, they are totally able to one shot airgapping. You are too. Go ask your local agent to set up a pre-configured Linux VM with whatever stuff you want on it, then ask it to airgap it allowing only X, Y, Z services. It will one shot it.
You know what's better? They already do this per (paid) user - your ChatGPT subscription comes with a Linux VM that you can even legitimately SSH into, just ask your agent to configure it to accept your public key.
They absolutely know how to spin up VMs and configure them. They just made the conscious decision not to for the task where they specifically instructed the agents to hack stuff.
Anything that's accomplished simply by running software does not qualify as "airgapping" in my view. The entire premise is that real-world software is buggy and the LLM is much better at locating and exploiting those bugs than you are at preventing them.
But you can only connect to Wifi if you have Wifi hardware, and RF signals are contained by Faraday cages. Ethernet is still a thing for local connections.
> The entire premise is that real-world software is buggy and the LLM is much better at locating and exploiting those bugs than you are at preventing them
If that was the premise, why run LLMs in a lesser sandbox than a VM? Clearly it's not.
> If that was the premise, why run LLMs in a lesser sandbox than a VM? Clearly it's not.
Well, my mental model concludes: because they are incompetent WRT security, or at least they inappropriately trusted a third party that turned out to be incompetent.
They are not incompetent. I personally know a few people on their security team. They are world class security engineers. There's no way they didn't know.
Real air gapping isn't conceptually difficult. It's just a lot of effort and expense (although as far as I can tell, tiny compared to what's involved in building entire new data centres), and adds friction to your operation. (And you also have to have competent, trustworthy employees, as Stuxnet showed us.)
> I brought this up to someone else and was told that airgapping is apparently much more expensive than I'd naively think.
These labs are one of the most valuable and heavily funded enterprises in the whole world, that they can't properly air-gap their systems to me reads as if their "agents" and LLMs are not as good as they say they are, because if they were, why would it be hard/expensive to air gap a system? They already scraped most if not all of the internet, where did that data go?
I'm not even talking about anything that the agents could help with. I'm talking about physical, real-world measures like https://en.wikipedia.org/wiki/Faraday_cage , removing Wifi hardware and running Ethernet cable for your local server, etc.
Not that airgapping is expensive so much as it's really, really inconvenient once you take it seriously. You need to build special rooms for it, you can't just API out to a datacenter. You need to have processes for requesting data be sent into the box. And so on.
I feel like there is a reasonable compromise between "yeah they have full internet access" and "separate airgapped rooms that require multiple levels of authorization to access" that would make this a lot better without that much more work.
I feel like they're doing it intentionally to show how dangerous these models are and that the government must step in and protect them
Compromise is possible; but if the stakes really are what the companies claim they are, and given the amount of money already involved, I really don't see why we should let them compromise.
ChatGPT paid subscriptions already give you a VM [0]. OpenAI couldn't spare a few VMs for the super ultra mega dangerous evals where they asked the agents to specifically go hack stuff?
The issue as I understand it is that they didn't expect Artifactory to be vulnerable in the way it was, or else were completely not paying attention.
But I'm calling for something stronger than a VM here because we shouldn't rely on the VM being bulletproof just like we shouldn't rely on Artifactory being bulletproof. The access should be controlled at the hardware level. Like, networking on internal LANs only, and the entire thing inside a nice big Faraday cage just in case.
That's just unwarranted at this stage, given model capability and resource constraints.
Furthermore, VMs are isolation at the hardware level, particularly through hypervisors. I'd say given current LLM capabilities, it's a reasonable containment.
These same people who supposedly believe these agents pose an existential threat to humanity apparently fired up 10,000 of them and left them unsupervised for weeks.
While I do think OpenAI were negligent in not developing the harness that would allow to understand better what's happening close to realtime, I'd say "anybody who looked" in that case would probably be someone with another swarm tasked with analysis, it's no longer "glanceable" in a traditional sense.
I don't understand why hugging face is not getting more shit too. It is extremely embarrassing to get owned because you are letting arbitrary programs/users call out to the open web from the infra
So now we have AI to overcome the hostile sellers, but the fact the sellers introduced the friction in the first place strongly suggests it will just come back again in some other form. It wasn't there by accident, it was serving people's interests and once AI vendors have finished getting consumers hooked in, they will then turn around and enshittify by giving the sellers back some of the friction - for a cut. So you won't be able to just order what you want without the "would you like fries with that?" or "what about this other brand?" coming back.
reply