I don't know, do we? What I do know is that it's much less a concern since toasters have no agency or ability to do anything. Unfortunately, LLMs do not share this trait. Not only have they been incurring greater agency and competence, but we've been just as happy to put them behind as many things as we can as they get better abd better.
So you tell me, is it smart to look at an entity that behaves like a concious agent and treat it like a toaster? Does that seem especially wise to you? If GPT-X orders a drone hit on you because it was, lets say 'quite upset' with your comments, will you cry out, 'It can't really be upset, so obviously the bullet in my head doesn't count.' Will God answer ?
It's amusing how silly these debates are. Constant quibbling about things that don't matter at all.
The biggest (but not the only) reason people afford moral consideration to other peoples is their ability to do something if those considerations are left unanswered.
Also, giving the abilities is fine, people should just be smart about it. Outward behavior is all that matters. I don't know that you are conscious, I simply strongly assume that you are. So don't create a machine that compels that assumption and model it like a toaster. It makes no sense.
If object a behaves like it has property x and you cannot otherwise test out x, then it is logical to assume it indeed has x. In this instance, 'object a has property x' is not an extraordinary claim at all. It is simply pointing out what has been observed. In fact, what is an extraordinary claim is saying what looks like a duck, quacks like a duck and moves like a duck is not a duck.
I imagine it will learn to win under some circumstances, perhaps in a case with some context expressing a desire to win. Drawing out an LLMs upper ability in the game should be fairly straightforward.
The weights are what need to evolve, and they certainly do during training. So yeah, emotions can happen by 'accident' as a result of the evolutionary pressure of predicting internet scale human text (amongst other things).
Weights, fixed by training, are not the same as emotions which are dynamic - innate systems detect inputs critical to survival (e.g. fast moving visual inputs, loud sounds), causing neurotransmitters like adrenaline and dopamine to be released, which then temporarily affect the operation of the cognitive system.
What you have in a pre-trained LLM is the ability to recognize emotions, and use that as one of the dozens of other context patterns it recognizes to predict continuations in the same style.
An LLM doesn't appear happy, sad, afraid, etc (to extent that it does - pretty minimal) because it is experiencing that emotion, but rather because it is predicting that it should appear that way. As people continue to anthropomorphize models, and take them at face value, this is a dangerous difference.
>What happened was mathematicians at openAI learned of an imminent development on this problem, and the insight that it entailed, then they were able to prompt a system in the correct direction and spend 20 million dollars to write down the final steps.
No it wasn't. They didn't know 'what direction' to take, and the solution they posted was not in fact the direction the authors took, so evidently you don't know what you're talking about. And they say LLMs hallucinate.
Frontier labs don't care about chess. If OpenAI cared, GPT-7 could be a grandmaster+ level chess player. In fact there's a google paper on grandmaster level chess without search with a 270M transformer. Outside that, there was gpt-3.5-turbo instruct which was incidentally a 1800 lichess elo player that didn't make any illegal moves even after a few thousand moves.
Frontier labs care deeply about automating knowledge work and computer use. They are working hard on getting models better and better, and they are succeeding. Astra is a step change on that front. So good luck i guess, if chess performance is your barometer.
> Frontier labs don't care about chess. If OpenAI cared, GPT-7 could be a grandmaster+ level chess player.
If the models were actually intelligent, the way that the boosters claim, they wouldn't need to be tuned to play chess in order to be good at it. That's kind of the point of intelligence, that it is generically applicable to whichever task one wishes.
Humans don't need a lot of training and finite tuning to make only legal moves.
An intelligent adult could simply read a short summary of the rules of chess and then, if they were careful, play a very bad game of chess without making illegal moves.
An LLM that has not been trained on any chess data cannot do that, at present. If you doubt it, take a current model and tell it that you want to play it at a variant of chess where, say, knights can also move diagonally like bishops. A human can easily adapt to this new ruleset (even if they make tactical mistakes, not having practiced with this variant of the rules).
How long a prompt do you think would be required to cajole an LLM into making legal moves at the rate of a human? Or do you think no amount of prompting could do that?
I don't know. My understanding is that current models will eventually fall into making illegal moves in longer chess games, and that no amount of prompting reliably gets them to stop doing so.
More importantly, beginner human players don't exhibit that tendency. The history of the position doesn't bother a human (except as required for castling and en passant rules), and the analysis becomes generally easier as pieces come off the board.
Humans do make these errors when playing blindfolded. If you even the playing field and give the LLM the position at each turn, it does not make mistakes.
Nothing is forcing the LLM to play 'blind'. If it's smart, it should be able to create its own representation of the chess board and update it with every move, just like a human would. Any chess engine that's sensitive to how the moves are formatted is clearly not very capable.
A human wouldn't do that, they'd look at the board. I'm not disagreeing that to demonstrate clear superhuman ability the LLM should be able to do this, but it plays better than most humans blindfolded, and with fair prompts seems very good otherwise.
That's what a human will do if they already have a physical board to look at. But if someone, say, posed you a chess exam question via FEN notation, or as a sequence of moves in algebraic notation, you'd sketch a visual representation of the board off your own initiative to help you answer the question. There is nothing in principle to stop the LLM creating its own board representations in whatever format enables it to easily keep track of game state and legal and illegal moves. If it fails to do so, that's a sign of its own limited understanding of chess as compared to a human.
The LLM would only be playing 'blindfolded' if you somehow forbade it from making notes (as you effectively do by literally blindfolding a human, given how limited human working memory is). But you are not doing that. The LLM is free to keep track of the game state via whatever means it chooses.
None of this is about superhuman ability. Any human who understands a given chess notation can convert it to a visual representation of a chess board and then use that representation to choose their next move, with their usual level of performance.
I maintain that the amount of effort to teach a human to do this vastly outweighs the amount of effort to teach an LLM to do this unless you're deliberately trying to make them fail. I honestly have no bigger point than that, I just think this isn't a very good thing by which to evaluate LLM capabilities. If there's no argument you'll accept, I am happy to move on.
You don’t need to teach a human anything except the rules of chess and the details of a particular chess notation. No special skill or training is required to make a sketch of a chess board. Surely there is no chess player who, if confronted with a sequence of chess moves in algebraic notation, would not think to construct a representation of the chess board in order to understand what was going on.
> I just think this isn't a very good thing by which to evaluate LLM capabilities
I don’t think any single task is a good way to evaluate LLM capabilities, but I don’t see why chess is worse than a lot of other tasks. (Of course it is of no practical consequence whether LLMs can play chess, so if you are just making that point, then yes, I agree.)
> If there's no argument you'll accept
It’s a little unfair to suggest that I wouldn’t accept any argument whatever for your position just because I haven’t been convinced by your very brief comments so far. I could equally well say the same thing to you!
Pretty much this. Feed it a book or two on chess, and you should have a decent (or good) player. That's the generic intelligence people have. The aims is not to be supremely talented at something, but being able to read a manual and figure how to use/play something. Mastery can be gained overtime.
If you gave a human a book or two on chess they would not become a decent player (they would be closer to 500-600 than 1100 ELO) and they would only get better after playing hundreds or thousands of games (often making illegal moves and moves that violate the rules of chess as they learn).
Your assumptions/intuition about generic human intelligence feels quite incorrect, considering LLMs currently play better than a brand new human player would (presumably without any attempt to fine tune them specific on chess, such as playing thousands of games).
That's quite untrue. I taught my (adult) brother the moves, the only illegal move he ever tried against me (over his 6 first games) was a castle with a rook that already moved twice. Within a few hundred games (less than 500 for sure, he played 3 minutes blitz but always took at least 10 minutes analyzing his games) he was rated 1100 on lichess (which is like 1050 on chess.com and unranked in the real world).
So your brother tried to make illegal moves while learning the game and it took your brother hundreds of games to get to be a decent player? I don't see how this contradicts anything I said...
The _only_ illegal move a human might make as a beginner is a failed en passant or a bad castle. And yes, a few hundred games is all it takes to be better than any publicly available LLM at the moment.
the discussion isn’t really about whether language models can become strong chess players though, the point is they seem to struggle to consistently make valid moves. Most humans don’t need to read two books to pick that up, just a couple lines of basic instructions
That has not been my experience with new players, they regularly make invalid or incorrect moves even after detailed instructions especially in novel situations.
Maybe it depends on the person? My six year old isn’t great at strategy but they can pretty consistently make valid moves. Sometimes they ask for confirmation on a move which is also not a trait I see in language models (at least unprompted)
Yes, but my point is that humans can’t even do the thing that the above comments are claiming humans can do (read a book or two and be decent at chess), and then they complain that LLMs can’t do the same thing (that humans can’t do either).
We seem to be moving goalposts to the point that humans don’t even live up to the expectations of the AI critics. The only way you get better at chess is by playing a lot of games and learning from mistakes, that goes for humans or AI agents, not simply by reading about chess.
> The only way you get better at chess is by playing a lot of games and learning from mistakes
How can you play without being aware of the rules and how can you learn from your mistakes without knowing they are mistakes? That’s what I said about reading a book of two. It is to kickstart the process. Then mastery is gained over time through practice.
This kickstarting then gradual refinement is how most people learn. And the foundational knowledge stays. Even a basic player knows to not do illegal moves.
Reading can kickstart the process, but you can also make random moves guided by some sort of system (such as a computer GUI) or learn by watching other players play. The overall point is that you learn through observation and lots of trial and error (whether you are a human or a computer). And beginners in chess often make illegal moves even after learning the rules, it's fairly common.
It feels like you're trying to say that humans never make illegal moves while learning chess, which doesn't match with my experience. I'm trying to understand your overall point.
> The overall point is that you learn through observation and lots of trial and error
That’s the most inefficient way and people usually avoid doing that. Instead they find someone that knows how to do the thing and ask him to be a teacher. Or use a proxy like a book or videos.
> It feels like you're trying to say that humans never make illegal moves while learning chess, which doesn't match with my experience. I'm trying to understand your overall point
There’s learning the basic stuff (which is done after a few games) and there’s mastery. The thread started with the observation that even with all that knowledge (through content ingested in training), LLMs still makes illegal moves. Humans can be erratic, but they can constrain themselves to the rules for the task at hand after learning them.
The only way to master anything is lots of trial and error. Coaches and teachers can help guide you towards more focused trial and error paths but the student still has to do the lessons and put in the work of learning, and learning only truly happens through doing.
Humans are not perfect and make mistakes in learning even when they have memorized the rules. A simple example is new players will often move a piece, exposing their king to check, and a more experienced player must point out to them that they have made an illegal move (because a new player often has not encoded that pattern for looking for exposed checks because they're more focused on how the pieces move, not what that piece exposes.)
We're just going to have to agree to disagree here.
If humans were actually intelligent, they wouldn't need to train and practice to play good chess. I mean, what level do you think people without any practice or training are ?
Oh right. But if all you do is reading programming books you are an amazing programmer? Where is all the training and practice LLMs did to become so good at coding?
It's called post-training, typically through some form of reinforcement learning, and is a significant part of modern LLM development.
You have the first stage, pre-training, which is learning from next token prediction. That's where the model memorises a lot of facts about things and generally gets good at forms of writing. It's like reading a lot of books on programming and reading through a lot of source code. It's learning how to autocomplete code, essentially. Doing that requires a developing a reasonable understanding of code, but it's also learning how to autocomplete bad code as well as good, and won't make it a "good" programmer.
Pre-training uses a method called Cross-Entropy Loss to update the weights of the network.
Then comes post-training. This is where the model is trained against huge sets of example problems, like fixing a bug, adding a new feature based on a spec, etc. They are set the task and try to complete it inside a training environment. Once they're done, their complete solution is evaluated (either by humans, or by some separate evaluation model that was developed based on human feedback) and they are updated based on whether the solution was good or not.
Post-training uses a different method called Proximal policy optimization to update the weights of the network.
So these really are very different forms of learning, and mainstream LLMs are not post-trained to be good at chess. They could be. You could easily create a reinforcement learning environment that evaluated and improved their ability to play and win at chess. The result would be a very strong chess playing AI, something we know is possible because the strongest chess playing programs we have are neural network based, but it is not a priority for AI companies.
>Where is all the training and practice LLMs did to become so good at coding?
Coding is a matter of translating the natural language description of a problem to the code specification while keeping the semantics fixed (and imputing the unspecified semantics as necessary). It is not considerably more difficult than translating between two dissimilar natural languages. Chess isn't a matter of language translation, but a compute heavy game of finding the best move out of many possibilities with wide variation in the quality of each move. Chess takes directed practice and reinforcement whereas language translation does not.
LLMs (and Humans) don't get really good from programming books lol. The training and practice is the actual code they predict and learn from in the process of predicting.
Oh I see. So if someone just reads books AND actual code then they can become experts, got it. And by the way LLMs are also trained with probably hundreds of thousands of actual games not just books
OpenAI making the next model good at chess is not analogous to a human training to get good at chess. It is analogous to God creating Human 2.0 which now has increased chess playing ability. If LLMs were intelligent the way humans are, then the models that exist right now would be able to spend time improving themselves at chess and become good at it. They can't do this because they are not, in fact, intelligent.
>Humanity is about to enter the phase when we will be using things based on ideas no human ever properly understands. This thought … disturbing, somehow?
This is just normal though. We were building sophisticated bronze and steel tools long before any complex understanding of metallurgy or chemistry. Medicine is still the wild west.
I first wanted to say fire, even though it's a cliche, but then I thought that in antiquity we used like everything without anything that would qualify today as understanding. Also now we have a lot of stuff that we "know" it works based on complicated numerical simulation.
I think the most "understanding" we ever had was in the 40s-50s designing nuclear bombs with slide rules. It was the culture that produced the idea of psychohistory.
> Medicine is still the wild west.
Reminder that we have no idea how anesthesia works.
New models are constantly being trained, but they don't have to be constanatly trained. If OpenAI or Anthropic 'just' wanted to be a profitable business, they could ease up on that, but they're both racing to create a machine to can automate all or most human labour.
If they stop racing, the wave of open models will pass them and their inference margins will drop to zero. A realistic model of their operating costs surely must include ongoing training.
>If they stop racing, the wave of open models will pass them and their inference margins will drop to zero.
1. ChatGPT's ~billion weekly active users aren't going to give a shit about some open source model, and neither would most of Anthropic's Enterprise cutomers.
2. Open AI and Anthropic are in a race between themselves, not open source model trainers. There's a reason those models are consistently several months behind and often perform much worse than benchmarks indicate. In the first place, they're only as close as they currently are from the distillation attacks on Anthropic and OpenAI. If they slowed down, they would slow down too.
1. They absolutely would, eventually, if open models surpassed Ant & OAI's flagships. Keep in mind "surpassed" encompasses both output quality and cost-saving architecture innovations like DSA, which may not be possible to apply to old models (and may take advantage of new hardware!)
2. "They're only as close..." is not natural law. You really think open weights couldn't catch up to a fixed target if Beijing makes it a priority? And what happens to their valuations if they abandon the goal of building AGI? There is no strategic alternative to constant training for these companies, which is why they're, uh, constantly training.
1. Mainstream users don't care about benchmarks or whether some open weight model has technically surpassed GPT-X on a leaderboard. They care about whther GPT does what they want it to do. Capable Open source models already exist, and that hasn't caused ordinary chatGPT users to abandon chatGPT for them. Hell Anthropic exists, and that didn't cause that either. OpenAI still dwarfs Anthropic in the consumer space. Obviously, sufficiently large differences in capability can eventually matter like when Anthropic blew everyone away in coding at one point, but that's very different from saying OpenAI has to train a frontier model every few months or inference margins go to zero.
2. Nobody said anything about a fixed target. Not sure why you interpreted 'slow down' as 'freeze current models forever'.
>And what happens to their valuations if they abandon the goal of building AGI?
The capabilities these companies already have, combined with their growing userbases, revenue and distribution are plausibly enough to sustain trillion dollar businesses already. OpenAI is a company with a billion active users that has started running ads that reached ARR of $1 billion in the first 2 months and Anthropic is a company that hit $11B+ in revenue last quarter after a pretty massive jump.
Enterprises absolutely care about benchmarks (especially internal ones, but the headline benchmaxxed ones too), and the Ant coding thing is a great example. How long did that last again? A few months? Illustrates my point perfectly. Switching is easy. Why would a business have any loyalty to one text->text endpoint over another? The consumer market may be less responsive to quality, sure, but it is more responsive to cost which I mentioned. It's also just not as big.
Well, I'm not really talking about freezing models forever either, I'm saying that nonstop training is a necessary part of their business. I don't think slowing down is untenable, I just think it's silly not to expect & account for ongoing training costs. That's all my original comment meant.
I also don't understand why you think the open labs couldn't catch up to a given level of quality. If something's been done twice already, why can't a well funded team of experts somewhere else do it a third time? Sounds like wishful thinking.
Enterprises have loyalty, but an obvious step-change in ability will sway. No, i don't think open source will zoom past the frontier (the necessary step change) just because they put the brakes on. Like i said, there's a reason Open Source is perpertually behind. If frontier labs decide to stop chasing AGI, then what, open source providers will step in to bleed a lot more money without being able to distill from the frontier ? To what end ?
> To what end?
To chase AGI, of course! An "open" lab does not have to stay open forever.
Are you confident that a step change in open source is unlikely? The industry seems to disagree, considering billions are being spent on them and billions are being spent to stay ahead of them. Open step changes have happened before (eg R1, or heck, self-attention). By the way, AIs themselves are quite good at writing GPU kernels now.
>But an LLM has no mind to feel bad if it cheats without getting caught
All the interpretability research we have would not indicate that "LLMs have no mind". It seems to me you have a conclusion and are working backwards to justify it. I guess I just don't see where 'they have no mind' would logically follow 'they sometimes cheat'.
I have somewhat of an understanding of how LLMs are created/operate, and from that I am drawing conclusions about if they can be moral or not. My conclusions match reality at the moment, but maybe that's a coincidence.
reply