I’m sympathetic to OP but think this is a hopeless battle.
1 - The commercial demand for anthropomorphised models is already immense, pre AGI.
2 - There is an intellectual hunger to engage with robot minds on questions of sentience. This too will grow with AGI.
I expect that tension of godlike minds that seem to be biddable and ownable like slaves is going to leak back into human-to-human morality, regardless of where we land on how we treat AI.
There’s an interesting academic group in the UK already focused on the model welfare debate, they seem to lean in favour of AI rights. No affiliation: https://www.prism-global.com/
AGI vs narrow AI is just a matter of generality (robustness - removing the fragility of being good at some things and awful at others). So far the biggest markets for AI (LLMs) are coding and business automation, two areas where you very much just want something reliable and without fake opinions and personality.
There is a market for ChatBots with personalities (and it always amuses me that the inventor of the transformer, Noam Shazeer, saw this as the greatest business opportunity for them with his character.ai), although from what we've seen this can be highly problematic, and in fact China has just banned "AI girlfriends and boyfriends".
I'd agree to that statement, though the common application of it is to conclude that we are almost there, just need to improve "the models". I wholeheartedly disagree with that. The human mind works quite differently from an LLM. The latter is closer to a toaster than to a human brain.
As to point 1, we know of no other general intelligence other than human minds. In this case it would be nearly impossible to build a general algorithm that's not us.
This is not an accident. Language was not a separate thing from the human mind, it was a feedback loop from increasing capabilities. Language lead to writing, writing lead to an information explosion of digital data. Said information explosion has lead to a situation where language can now bootstrap itself and become a separate thing. We are and have created a biosphere for language based, um, life. With our sciences we have mapped out reality in language. Past molecules and atoms, past the particles that make particles down to fields. We have computer controlled just about everything. Every year the analog hole shrinks further.
I heard something to the effect of "Language has gathered enough information to escape its meat prison".
I call it the "Language as an SCP". It has a plan and we're not privy to it.
Simple recommendation app that asks 10+ AI models for the top options for whatever you’re looking for, and it pools the results to give you the overall top three:
Yes, all of the links seem to go through to shopping funnels, even to get a PDF. I’m not seeing any freely downloadable resources. This is basically just an ad.
A new and growing fear I have is that AI cuts off some of the most well-trodden pathways to intellectual growth. Every time the AI solves a problem that I would not have been able to efficiently solve alone, it has replaced an interaction that previously would have happened with a mentor/supervisor/code reviewer. This is faster for the person seeking the micro-assistance, but there is a flipside: fewer human-to-human acts of mentorship and shared problem solving. We lose a lively ingredient of team formation, expert formation, and a source of joy for all involved. As this scales up it feels plausible that we end up with a wider deagradation of intellectual standards. I like the Cognitive Comons framing.
I think this has already happened to some extent, with social media etc. crowding out reading books. Even higher quality stuff like HN pales in comparison with what a good book brings, and it's a daily struggle to keep reading.
At least we're (mostly) not scrolling feeds all day at work. AI is going to be like this for all our professional competencies.
If Hacker News is what passes for high quality content in 2026, I feel well and truly bad for the current generation who will never get to experience anything like the old school SMF forums, or pre-2014 Reddit.
>Every time the AI solves a problem that I would not have been able to efficiently solve alone, it has replaced an interaction that previously would have happened with a mentor/supervisor/code reviewer.
I suppose that Socrates would strongly agree, seeing how the same argument could be applied to learning from a book.
Reading a book is a form of communication between the writers the readers, and literary culture enriches this with the ongoing interactions around texts. The argument voiced by Socrates in the Phaedrus is not primarily that it removes or stifles interaction but that it undermines memory (and thinking) and that it will "create forgetfulness in the learners' souls, because they will not use their memories; they will trust to the external written characters and not remember of themselves". AFAIK, there is well established research that shows that the move from oral to literate cultures does indeed bear this out. More recent studies about how wayfinding deteriorates when we rely on GPS seems a like a similar dynamic.
There is another charge against writing in the Phaedrus, which is that texts cannot answer for themselves (contrary to spoken interlocutors) or adjust their communication to the needs and character of the reader:
> when they have been once written down they are tumbled about anywhere among those who may or may not understand them, and know not to whom they should reply, to whom not: and, if they are maltreated or abused, they have no parent to protect them; and they cannot protect or defend themselves
An interesting thing about computation is that it is a writing and re-writing that can indeed be responsive and, in a non-trivial sense, "answer for itself". LLMs do this in a form that is obviously very successful for engagement (and for hype) and for producing a lot of compelling output.
I think the Phaedrus also has something to say here:
> Soc. Then [they who who knows the just and good and honourable] will not seriously incline to "write" his thoughts "in water" with pen and ink, sowing words which can neither speak for themselves nor teach the truth adequately to others?
>
> Phaedr. No, that is not likely.
>
> Soc. No, that is not likely--in the garden of letters he will sow and plant, but only for the sake of recreation and amusement; he will write them down as memorials to be treasured against the forgetfulness of old age, by himself, or by any other old man who is treading the same path. He will rejoice in beholding their tender growth; and while others are refreshing their souls with banqueting and the like, this will be the pastime in which his days are spent.
Interesting, so based on this, do you think that Socrates would have been more ok with learning from an AI that can speak back to defend its claims (assuming we address the sycophantic tendencies) than with learning from "frozen" books?
I wonder! I do think that that the systematic rewriting realized by computation complicates the nature of writing in ways that radically deepen the problematic. But in fact, the development of literary culture more generally already deepened the problem, because, as we know now, "frozen" books come alive in the proliferation of secondary literature, literary interpretation, derivative works, translations, collage, etc. In my view, none of this solves the dangers of writing Plato recounted, but they make it more complicated and are obviously essential to the way our thinking and world civilization has developed.
Regarding LLMs specifically, my view is that the current state of LLM-based AI is leading us more towards '"write" our thoughts "in water"', because
* what we write to LLMs does not catalyze a transformation in understanding of them as systems, nor drive them to evolve in a sustainable way towards truth,
* and their outputs are the result of potent patterns in the turbulence of humanity's discourse, rather than an understanding or systematization that could "teach the truth adequately to others"
While I think their potency cannot be questioned, their sustainability and correctness certainly can be. IMO, AI in the vein of theorem provers and semantic webs are much more in line with a text that can actually speak back with integrity and genuine responsiveness, "teaching truth adequately". But I also guess
* the LLM profusion will likely catalyze some important developments and advances in those other approaches (and some use of LLMs will persist even after advances on a difference basis take over) and
* probabilistic methods are here to stay and will likely be more integrated into symbolic systems and
I am fairly certain Socrates knew about books, he just wasn't literate himself. But literacy was quite common amongst Athenian citizens, it was sort of an outlier for the time. Of course most of the day-labor was done by slaves, but it afforded them the time for cultural pursuits and healthy debate, and Socrates was very much a product of that.
If the LLMs work as advertised, we won't need experts.
Even today, with the current state of the art, we need far fewer experts. LLMs are already much better at debugging than humans are, for example. Math proofs are falling at a unprecedented rate. And so on.
In the future, we may not find mere human levels of intelligence so useful. Let consumers consume.
I mean why would you ask locals? There's a good chance that the person you asked is a tourist or an immigrant, and regardless of that, they'll all just... pull up their own phone and check on Google Maps, at the very least so they can show you the route on the map. Why would you ask another person to be a "meat proxy"[0] for the thing you can trivially check yourself?
--
[0] - Some new phrase I suddenly saw pop up everywhere in the last few days. Must be popular with cool kidz.
"In the future, everyone will be world-famous for 15 minutes" _in a microniche techno-linguistic community, at a time and choosing of the swirling AI clouds_
1 - The commercial demand for anthropomorphised models is already immense, pre AGI.
2 - There is an intellectual hunger to engage with robot minds on questions of sentience. This too will grow with AGI.
I expect that tension of godlike minds that seem to be biddable and ownable like slaves is going to leak back into human-to-human morality, regardless of where we land on how we treat AI.
There’s an interesting academic group in the UK already focused on the model welfare debate, they seem to lean in favour of AI rights. No affiliation: https://www.prism-global.com/
reply