An interesting talk I heard at a conference once, that I can neither remember the speaker for or speak to their legitimacy, suggested that we might have some lower form of intelligence encoded into our language. They posed the idea that we have enough unique words, and combination of words, that it starts to have reason unto itself similar to how our neurons and their connection breed intelligence. The idea was that we as humans have baked intelligence into our own speech patterns. It seemed a little to abstract for me, but potentially goes a little way to explaining how a statistical averaging algorithm with some randomness, at scale, starts to look like it very occasionally has a genuinely novel thought.
In The Ticket That Exploded, William S. Burroughs proposes language is a virus in itself, coming from the Outside, and infecting the host with it's control logic. In Radio Free Abemuth, Philip K. Dick attributes a similar possession to a benevolent force, akin to the divine Logos flourishing intelligent development. Both seem open to an impersonal agency that maps to intelligent systems encoded in their transfer protocols.
The english language pattern is definitely shaping our thoughts and limiting our ideas. Just the whole idea of going from some abstract thought to actually making it out in tangible language, I mean no matter the thought it's a lossy transfer into a medium that lacks all the dimensionality of subconscious thought, neurotransmitter action, sensory information, and physiological response.
There's also something to consider with lower level vs higher level abstractions in language. E.g. jargon. One short word could have a 200 page thesis behind it defining all the ramifications. Talk about compression of information.
Now imagine if our language lacked say the mechanism of jargon, of using some meta word to define thousands of stringed together words at once. Every idea like "car" would have to be described from first principles. The species would probably never develop technology with this sort of language pattern present. If we could somehow level up beyond our current abstraction level, maybe that would make us even smarter, able to handle bigger ideas quicker in real time.
Even more simply than all this: I can only speak about what I have english words for.
If you are fluent in a second language, you understand that the only way to become truly fluent is not just to learn the tangible aspects - the vocabulary, grammar, references, expressions, etc - you have to learn to think like the language/culture.
It’s a combination of cultural assumptions, facial expressions and affectations, thinking patterns, and a whole cultural upbringing that can lead you to very different mental processes and natural conclusions starting from the same words and phrases.
Language absolutely encodes a certain form of intelligence. A lot of those things are reflected not just in the totality of the culture but the language itself. Being fluent leads you to different thinking patterns and different conclusions when processing in that language.
Absolutely? I don't think it does so much. People speaking different languages seem to have very similar thoughts. It's true that fluency is a wholistic performance, but I don't think there are any particular thoughts that can't be translated.
At one point there was a universal language, and later either recreated or scrambled, allowing for humans to have many languages for increased confusion.
Now that we are once again attempting to unify our language we find ourselves in a pursuit to build something to escape the Earth.
I am speaking of biblical sources as reasoning or evidence. If you read different books and hold a different faith that would explain the missing part.
I thought that was how most people understood LLM’s capabilities? We have spent millenia creating language to map onto our world. Therefore, implicit in that language is a simulacrum of our world.
I feel like I can feel this happening in my mind in real time. Something like: the part of my brain that thinks thoughts is fairly rudimentary, basically just impressions or hunches--but then there's another part which translates them into words and grammar, and when it takes an impression it can translate it into something fairly sophisticated and intelligent, because it's somehow necessary in order to create a sentence which actually captures the impression.
Does that mean the language(s) we speak determine how intelligent we are? Could learning French, for example—often considered a more expressive language—make a native English speaker more intelligent or even more compassionate?
Does that explain why different countries that speak different languages have different engineering cultures? Like is german better suited towards engineering than english for example?
Interesting. Would this apply to any rich enough system of expression, like music or art? Or is there something specific about language that makes it different?
It's pretty clear reading from these comments that most HN members have a 2023-era impression of LLMs.
Modern chain-of-thought models with RL post training on verifiable tasks + realistic environments + rubrics are worlds apart from models trained on a simple next token prediction objective.
More money goes into the rubrics and RL environments than individual training runs themselves.
(Yes, at inference-time LLMs still output words one at a time, much like human speakers. But don't confuse the mechanism with the training objective.)
Even with heavy RL post training and rubrics, the model is still fundamentally bound by the next token prediction mechanism at inference. Rlhf and cot just affect the probability distribution of which tokens get predicted next. Take away the heavy agentic scaffolding and external feedback loops, and a single hallucinated token can still derail the entire chain of thought.
But but but....I was told it was a stochastic parrot! I liked that idea because it appealed to my vanity, and it described the gibberish produced by older models with bad prompting, and that was enough for me thank you.
Nobody knows how it works, really. It just turned out that if you try to predict the next word then you get intelligent behavior, depending on amount of training data, and the size and topology of the network. But again, nobody knows why, and what the limits are.
Agreed. We went this direction for our golems, djinns, and other mechanistic minds because we believe it sort of reflects the primitives of our own neurons (which we also don't fully grok).
I heard someone who studies this sort of thing say basically what biological neurons are trying to do is predict as well. Predicting what exactly? I’m not sure. The next time they should fire or something. I can’t find the YouTube video now.
In the last few decades, there has been an increased interest in the role of prediction in language comprehension. The idea that people predict (i.e., context-based pre-activation of upcoming linguistic input) was deemed controversial at first. However, present-day theories of language comprehension have embraced linguistic prediction as the main reason why language processing tends to be so effortless, accurate, and efficient.
Predicting reality, under the "controlled hallucination" framing, corrected by sensory error signals. The brain has no access to ground truth, only input data that helps correct the hallucination.
Based on lots of human interactions, I think there are a lot of human beings out there who mentally aren’t much more than “next word predictors” who happen to be made of meat+neurons instead of silicon+code.
It's more accurate to say we grew them. That is the breakthrough of Deep Learning. We left the hard part to the machine (learning how to do what you want it to do) to figure out during training.
And that means we are not privy to whatever things it has learnt in its trillions of weights.
Right, but what's the limit of what you can deduce computationally from truly vast training sets? How much structure is there in the subtext of what's written down? It looks like there's rather a lot.
Omniscience doesn’t imply that you have to store all the information, but that you can retrieve/reconstruct it, and reasoning allows it. In fact would be impossible for any physical intelligence to store all the information as plain as it is infinite.
I'm not an expert, but my current mental model for this sort of thing is that the thoughts were already there, somewhere in the training data.
Some human was looking for something like this once. They didn't find it, but they wrote about the search precisely enough that the finding can happen during inferrence.
Maybe somebody will come along and school me, but for now it's a fun way to think about it: A million dead ends, each with a uniquely disappointed human, now with a chance at a second life in the hands of a different human they haven't met. If only the weights had encoded enough to introduce us, supposing they still live.
Intelligence as a measure of the ability to define predictive models of certain problems (and their solutions).
Promoting LLMs is encoding the problem we want into the query vectors, and through the magic of the complex training and the power of operations in a very large dimensional abstract space the AI can manipulate the representations, and iteratively approximate solutions. (And using bigger and bigger contexts and better encodings it can form better models.)
Here's my grok of it: Deep learning models progressively abstract a concept presented at the input by passing the input through many sequential layers () until an output layer transforms the output of the final layer into something interpretable, such as an indication of what token to predict next, or a classification, or whatever. The transformer architecture futhermore offers layers that allow different parts of the previous layer's output to sort of mix with each other in complex ways. As you get into greater levels of abstraction, the attention process is mixing very abstract concepts with each other in a nonetheless highly structured manner. I believe this is where the intelligence lives.
sometimes with residual connections, but we can ignore that for sake of simplicity.
I don't. I seem to think at a more abstract, pre-verbal level rather than through an internal voice.
Some studies suggest that frequent internal monologue may occur in roughly 30–50% of people [1], but the research is based on relatively small samples.
Theres more than words in our minds. Think harder are you absolutely sure? You REASON with words but your ideas dont form just from you reasoning. The ideas just seem to come out of nowhere to the part of your brain that then reasons around them.
I tend to agree with Albert Einstein below; there's a very physical/spatial aspect to my problem solving before it can be translated to words. I work in software so there's nothing innately physical about it. Never put much thought to it until LLMs brought it up for debate.
"The words of the language, as they are written or spoken, do not seem to play any role in my mechanism of thought. The psychical entities which seem to serve as elements in thought are certain signs and more or less clear images which can be "voluntarily" reproduced and combined....From a psychological viewpoint this combinatory play seems to be the essential feature in productive thought....The...elements are, in my case, of visual and some of muscular type. Conventional words or other signs have to be sought for laboriously only in a secondary stage, when the mentioned associative play is sufficiently established and can be reproduced at will."
How do you know that it's the words driving the thinking, rather than the stream of words just being an observable trace tacked onto the actual thinking?
These days, words. When I was in an environment where language swapping between 4 to 5 languages was common, I thought in pictures and described it in the correct language for the audience. It was a plasticity mind trip.
Also saved pesos on the charge-per-text SMS schemes the local phone companies used because we could embed information across so many options.
You think with and without words. When you have to pee, it isn't like you speak to yourself "Gee, pinch in the loins, I guess that must mean must have to pee. Alright legs, get me up off my butt. Left right left right left right. Stop. Hand, get the zipper going. Johnson, your turn now."
Not OP but I have this too. It’s like a voice in your head constantly. It can make writing very easy as you just transcribe the internal monologue.
It’s weird, I’d be hesitant to say “it’s a voice” but it kind of is and it is not my own which I find curious (who on earth is speaking in my head). In some ways it sad, if I close my eyes, I can’t picture a sunset and I can’t really dream. I love reading books but I can’t visualise the settings properly but it resonates with how my mind describes the world to itself.
AI is way beyond conventional LLM architecture now. It combines LLMs with search + RL. The traditional LLM architecture hit a wall around GPT-4o. Arc AGI evals show this.
All that extra is clear as day compared to the mystery of how neural network training decides to divide and balance the weights in even small neutral networks.
We can, at best, approach a good set of weights, even in tiny neural networks.
Imagine if we found a way to calculate the exact optimal weights for a given loss function. I mean, there is an exact optimal solution, it exists, but we can't find it exactly, even for a neural network with just 50 parameters.
There is no point in that because the loss function itself is already an approximation. No one knows what is the exact loss function for any given non-trivial real-world task.
I mean, things humans defined can be pretty clear. Like your electricity rate. Natural systems less so. Not pretending no complexity in human made things, but at least some models can be fully specified.
Most human reasoning happens within language - even mathematics is an abstraction that allows us to map concepts we don’t natively hold into a linguistic processing layer.
Language emanates from intelligence. That means the patterns and structure that make up human intelligence will appear in language. LLMs are created through so much language training that they can approximate (and now to some degree exceed) human intelligence using pattern recognition, statistics, and autocomplete (in layman’s terms).
Not sure how it is now, but early “reasoning” was simply the big labs sticking “wait a minute, what if I…” type language blocks into the process to trigger something like our own internal reasoning.
I don't think LLMs currently have direct reasoning abilities, but as we make them more complicated (MoE, RL) I think we're getting better at learning an implicit world model that guides the token output distribution towards making good hypotheses.
I mean, the subtlety of the neural network weights that emerge from training are not fully comprehended by anyone, man or machine.
Every individual calculation is understood, and every step of training is understood, but the exact nature of those weights that divide the responsibility of responding to subtle changes of input in intelligent ways is beyond me.
Stephen Wolfram had a great description of this effect in the early days (GPT 3.5 era):
Machine learning trains the network to do... anything that you reward it for. If you keep training, it keeps getting better.
Next word prediction can always keep getting better.
At first, simply "learning" spelling is what makes the predictions better because tokens are word chunks, not always whole words.
Then, the models "run out of steam" and can't get any better by learning more spelling rules, but the gradient descent forces them to get better... so they do... by learning the rules of grammar.
At this point the AIs can output correctly spelled and grammatically coherent sentences, but the sentences ramble on about nonsense topics.
So what happens next as the models run out of grammar rules is that they're forced to learn the rules "above grammar": logic, world knowledge, coherent story telling, etc.
At some point they learn to output pages and pages of fluid, coherent text, but... if they're not smart, if they don't think, and if they don't know what they're talking about, then they're still "suboptimal" and their forced gradient descent will make them close those gaps.
Eventually, the only way they can improve at "next token prediction" is by building up to human-like intelligence, including an inner monologue, theory of mind, and everything.
If I were to guess, being pleasantly surprised is just a learned appropriate social response from the expectation of receiving a reward and as such, that social norm is codified sufficiently enough in our writings that it appears in LLMs output.
It’s sort of like all the people who will ask Claude or GPT to validate their complete nonsense and receive unyielding praise for it, the models just learned that this is the best received response based on training data and RL.
I bet these same sorts of expressions can be found in practically every failed attempt as well.
EU was very clear that closing down the OS (Apple) is how they avoid fines with their latest rulings. They punished Google for open sourcing the OS and left Apple get away free with explicit explanation that closed OS doesn't need to adhere to same rules.
kremlinology is the term for any political intelligence gathering that's all second-order. In the canonical case of the Kremlin it's because it was very difficult to get inside data and even then there were so many moving parts that you needed an expert in just that area who had alot of data like who was having drinks with whom to predict who might be on the out and who might be ascendant.
There was one analyst reading Pravda who noticed a lot of people were suddenly being promoted into positions held by members of a certain large clique, with no notice about what was happening to the former appointees.
After one of the nuclear treaties was signed, there was a guy whose photo was published in Pravda, shaking hands with Gorbachev; a different fresh photo of him shaking hands with Gorbachev every day for an entire month. He turned out to be the head of the ministry of strategic deception.
kremlinology is an old pre-1989 term for people trying to observe and interpret the workings of a tightly sealed institution. Could be used for Vatican rumors and news, too, for example, or North Korea, etc.
Similar to that idea of trying to derive stuff from an indirect channel, but much more distant. (There is an obvious link between food consumption and building occupancy.)
Stuff like watching the order of names on public lists of attendees for dinner events, etc. If someone disappeared or moved off the top few names, perhaps they're in trouble or have been purged from the party ... or maybe they're just sick this week.
A similar hack created The Panama Papers. Not much came from those either. Like the Epstein Files, it may make for interesting reading, but I don't expect much legal action.
LLM use language, but it can't "think" about biochemistry
I saw that LLM have reasoning capabilities, which is different from machine learning, but I don't understand how it works.
reply