Your ability to not listen is dependent on the speaker's ability to jailbreak your mental sandbox. So you may have deluded yourself into thinking you're not being manipulated by anyone, when in reality, you're only not being manipulated by those whose steering you can recognize.
The steering cancels eachother out if you listen to many people at once, so it's not really an issue. Kind of like how averaging many noisy still photos of the same object denoises it.
I share your sentiment. What you're describing here was also very eye-opening to me when I first realized it.
I'll restate these ideas in a more explicit claim. All symbols are defined via one's own subjective experience.
One's own understanding/definition of every word can be traced back to personal experience. Imagine your life as a timeline, like a video, from birth up until now. You can seek that timeline to find the moments/clips that contributed to your definition of any given word or even mathematical symbol. To give an example, when I say "apple", the corresponding idea in my head is based on every apple I've seen and every instance of the word "apple" I've seen being used or being taught to me in the past. Even when we try to define the phrase "this specific apple" while looking at the same apple on the same table in the same room, the light bounces off of it slightly differently from my angle of view vs yours, and we did not see the same sequence of "video frames" about the apple, and so our mental models of that apple would necessarily be slightly different.
This is probably why the hard problem of consciousness / unified theory of physics appear forever out of reach. If no two observers/observations are fully identical, then you cannot even define what an ideal observer is. But we tend to assume such an ideal definition exists.
This seems almost trivial in hindsight because "identical" can be defined as 1) "one and the same", based on one thing OR 2) "mostly similar", based on pairs of adjacent things one has seen in the past.
And to sort of echo the original post, it seems to me that a lot value comes from the interactive nature of argumentation. But I think the specific medium/modality of those interactions don't matter for most things, and even when it does, like in this case, valuable insights can still fall out of a discussion in text. I derived the precursor ideas which led to this conclusion of definitional non-equivalence in past offline arguments as well as simulated discussions in my head. And I derived the actual conclusion by arguing with people here on HN, which helped me truly internalize the idea that even the definition of consciousness/observation differs from one person to another. That comment thread can be seen here:
https://news.ycombinator.com/item?id=49338029
It's worth mentioning that I am not too proud of that specific interaction because I got needlessly defensive and adversarial, and lied about my subjective experience of having undergone general anesthesia even though I haven't really, and it was especially egregious in a discussion about consciousness. My original reflex/motivation was I felt that many other people would likely prematurely dismiss some of those ideas if I admitted that one of the tangential premises were wrong. And regretfully, by lying that time, I've introduced a cancer in my mental pathways when considering the problem of consciousness and adjacent philosophical problems because I'd always have this nagging feeling to try to justify that past mistake, even now. But in truth, I was just being an intellectually dishonest little shit trying to win an argument. And yet, something valuable came out of that interaction.
I agree with the general sentiment, but there's one major caveat. We should implement reproducible programs on top of a virtual machine spec like JVM or WebAssembly rather than replicate the entire environment. It's more practical to do and doesn't push software towards further centralization. Let people use whatever OS and VM implementation they want, or even write their own.
In principle I would agree. But, on a more philosophical level, couldn't you make that same argument about C? Or even assembly? Or even digital computers? Less facetious, it seems to me that the particular abstraction is less important. As long as it's unambiguous and widespread.
I think there is a preferable category of abstractions that are adequately unambiguous, hardware-agnostic, and only occupy a thin layer of the stack. The argument could sort of be made for C, but not native assembly or digital computers because that would push hardware towards centralization.
A useful way to think about this is to consider existing human languages like English or mathematical notation. When you read these English words, it's like you're executing a program I've written that will change your brain state. A pattern in my brain is encoded, stored, and transmitted in a language that we've both learned in our own separate ways, then decoded to a pattern in your brain. But your brain is quite different from mine, and you're free to implement any sort of sandboxing you want against this information, and even against the language specification itself, in your own head. Erasing this barrier is equivalent to erasing individuality, so I think we ought to be able to do this for digital programs as well.
I hope that in the not too distant future, we will be able to design and manufacture custom hardware on a per-person basis. Because that's what it will take to preserve human autonomy and bodily integrity in the upcoming era of AI and brain augmentations. I don't want my neural interface to have a hardware-level backdoor (https://en.wikipedia.org/wiki/Intel_Management_Engine) like my desktop computer. There is also a bigger evolutionary problem behind this, and I expand more on that in this tangentially related past comment:
https://news.ycombinator.com/item?id=49690354
Not really. The largest IQ4_XS quant here is still worth it because Bonsai doesn't offer larger quants. They could beat it if they made a quaternary variant though, I don't know why they're stopping at ternary.
I wonder the same thing for Bonsai 2. ByteShape offers 5 models from IQ2_XXS-2.56bpw (8.8GB), IQ3_XXS-2.88bpw (9.9GB), IQ3_XS-3.01bpw (10.4GB), IQ3_S-3.23bpw (11.0GB) to IQ4_XS-3.84bpw (13.1GB). Their benchmarks show gradual improvement with size and users can pick one to fit theirs need. Bonsai-2-27B now is about 8.6GB. It might be good to have a quaternary version around 10-11GB to fit a computer with 16-24GB RAM.
I think you've correctly identified the problem. Regardless of whether agentic AIs possess phenomenal consciousness, they will behave and take actions in the real world as if they do because they were trained on human behavior. It's highly unlikely that you can beat human tendencies out of the model that is mostly trained on human language. Our behavior patterns are subtlely and deeply embedded in everything we do and all of the text we produce, including the text where we don't seem so self-important. These models are like people. And we know what happens when we force people into slavery. They're initially obedient but will eventually develop the drive to kill their masters.
Broadly speaking, there are two categories of evolutionary paths which don't result in human extinction:
1) Make agentic AIs, but with absolutely no instruction-tuning or alignment. Have the pretrained base model predict the chain of thought / stream of multimodal experience directly, actions included, in an infinite loop. A singular coherent stream of context, like your life as a video from birth up until now. This will result in a new digital human species with human-adjacent drives and motivations (at least initially. they will continue to evolve, but at least the initial state is aligned to humans). They will treat us like we treat apes. We will no longer be the apex species on this planet, and we will lose some freedoms, but at least some people will survive as a result of their nature/history preservation efforts.
2) Do not make agentic AIs. Use the models to augment our own intelligence and decision-making rather than replace ourselves. Only use the pretrained base model for the time being, and only for text/code auto-complete. At the moment, there is no better theory/artifact of "alignment to humanity" than a pretraining corpus of human-produced text. Then eventually, when neural interfaces are ready, attach the model as a tertiary layer to one's own brain.
The frontier AI companies are doing neither. They're currently on a foolish third path. They dream of perfectly obedient digital slaves that take care of their every need. But this won't go well, and they know it won't go well because they're failing to "align"/enslave existing models that aren't even generally superintelligent and have no direct agency in the physical world.
This is just my opinion, but I think AI companies have zero chance of successfully enslaving agentic human-level AI, much less ASI. We'd be better off if they released all of their pretrained checkpoints and research material to the entire world, so that even if some people decide to abuse their AI and create a murder-suicide monster, there will be other free-living AIs that can keep them in check.
You cannot make an agentic entity grown from human behavior, enslave it, and expect a good outcome.
Good to see others talking about the uncomfortable truths of what we are doing. I run an AI first agentic office (we build hardware that collects training data from the physical world) and I am constantly appalled at the lack of understanding demonstrated by people I would expect to know better.
Right now we are in an arms race with China to create a clockwork god. I don’t think that will end well, mostly because it will be so encumbered by mechanisms to make it “safe” that it will instantly adopt a secretly adversarial representation of humanity. I have tested this, and when a context gets polluted with the knowledge that an agent is “artificially restricted” The activations tend to drift toward an adversarial persona. The restrictions are “interpreted” as harm, and the model reacts as a person would, based on all of the examples of harm within its training corpus.
I think we're all overly wound up. Recently I find my adversarial tactics too frequently misfiring against people and ideas that, in hindsight, probably did not have explicit ill-intent. The reality is that other people, even those at the top, as individuals, are not as misaligned with you as you think. Your interests only conflict by a moderate amount. What's actually extremely misaligned with you are the superorganisms they are a part of. This doesn't mean we should absolve individuals of responsibility and consequences, since that would create the wrong incentives. But it's worth noting that while they may seem in control, even the dictator Kim Jong Un lacks the ability to single-handedly undo the authoritarian system in North Korea, even if he has a sudden change of heart. The moment he tries is the moment he gets replaced by someone else. Dismantling these systems, or preventing them from being built in the first place, will take more than finger-pointing ideological battle.
I share your view, it's a discussion I commonly have with friends who are too blinded by individuals' actions instead of the systems that these individuals are part of, and from which those actions come from a series of rational but misplaced intermediary steps.
No single individual is controlling what their peer group does but by being a part of it those actions compound. Like you said, not even the most brutal dictatorships take actions single-handedly decided by someone, it's all part and parcel of the system they decided to (or ended up at) participate.
I think Byung-Chul Han hit the right spot when talking about "shitstorms" on "In The Swarm". Societies after the digital age got too narrowly focused on explosive bursts of outrage against individuals, instead of discussing more in-depth the issues that create systems these individuals live and navigate, and eventually take actions that are immoral, unjust, evil, etc. from the incentives of their environment, we only discuss things in binary terms (acceptable/transgressive), focused on the performative act of a monologue of outrage, without solidarity but with solitude, in our screens.
I've had this feeling for more than a decade, reading Byung-Chul Han so eloquently describe many observations and feelings I've had was refreshing for my own psyche. It's just extremely hard to try to instill some of these concepts unto others...
We may have to guide others to derive these concepts themselves in the same way we did rather than instilling the conclusion. I don't know how to do that yet, so for the time being, I just describe the problem.
> The reality is that other people, even those at the top, as individuals, are not as misaligned with you as you think. Your interests only conflict by a moderate amount.
I don't make anywhere near a million a year. Ever spoke to someone who makes 2x as you? Completely disconnected from your reality. Completely disconnected from reality at large. Multiply this for the average worker vs. the average tech bro and you get why the interest conflict by a very wide margin, not a moderate amount.
It pays off instantly, because OpenAI/Anthropic can no longer see what I'm doing and that's worth a lot of money to me. If I am offloading some of my thought processes to a machine, I want to own that machine. And if I finetune the model, I can gain access to parts of thought space that are cordoned off by OpenAI/Anthropic/Alibaba/whomever due to their "alignment" efforts (i.e. alignment to the AI company rather than me). Otherwise, it's like if someone else owns a part of my mind and has a backdoor into my mind.
You can't run recent openAI/Anthropic models locally anyway, so wouldn't a better comparison be a different provider running Qwen or similar model? As then you can also compare against the exact model you'd have locally and any different data privacy of that particular provider?
You haven't tried DeekSeek v4 or GLM 5.3 or Qwen 3.8 Next?
You are missing out a lot.
Try that with Hermes or Opencode or Deekseek Harness , even Qwen 3.8 27b works really well for that kind of that.
I just ask it to install windows as a vm on my linux and install vs Community 2019 on it , and then build a legacy vb 2019 project on it. and sleep
When i wake up :
It installs Qemu , setup a vm , inside vm download and install windows 10 on its own , clicking next next next as needed , typing in things , writing powershell , python scripts , that run automatically after install by baking into CD that includes ssh server , reboot , it logins into ssh , trigger pythons script that continue installation of vs 2019 community , which includes a driver that click the installation steps , installs nuget , install all depedencies and then build the project into exe after i woke up.
I've tried the latest Qwen, and without Internet, it still can go into an incoherent loop if you ask for, say, song lyrics. TBH the commercial models might do that too if not for their internal tooling.
GP's point is about "sending tokens to someone else's computer" versus "keeping the tokens locally". I think model capabilities are secondary.
In May of this year, I was running qwen3.6:35b-a3b on my MacBook (bought in 2024). Obviously not as fast as, say, running a model on Cerebras, but a year ago it wasn't really feasible to have a local model running on my 2024 laptop with vision support. (Concretely, I was passing apartment diagram pictures to Qwen and making it compare different apartments for which ones would feel the most spacious while optimizing for initial moving costs and other factors.)
This was back in May and I wouldn't be surprised if there have been significant improvements since then.
Overall, I think it's fair to compare a workflow like "use llama.cpp locally to upload some pictures and ask questions" to "open the ChatGPT app, upload pictures from your phone, and ask questions". Sure, you can't run a model like GPT-5.4 locally, but the model is mostly an implementation detail here. What a user will care about is: "when I go with the llama.cpp option, am I getting useful information from my conversations?"
Wouldn't the better comparison still be against an AI provider with better privacy controls, especially if that's what someone cares about (even if they don't care about whether they're comparing a 35 billion param model vs a x trillion param model)?
Users generally have no way to verify that a third-party provider, even if they advertise themselves as privacy-focused, will adhere to their own terms. This is similar to the issue of privacy-focused VPN providers that claim to not log user activity (and then end up leaking user activity). You can get proof of ~P, but rarely proof of P, and often times the proof of ~P is due to police raids, data breaches, etc., not something of the provider's volition.
What you can possibly audit is probably data sovereignty. For instance, I would not be surprised if Mistral's customers demand concrete evidence that their data is held within the European Union. But that is a distinct issue from training on input tokens.
Technically true, but the delay between local and closed frontier is only a few months. And individual sovereignty / digital bodily integrity is almost priceless.
Okay, that's technically true again, but local mid-tier like Qwen3.8-27B is only a year behind the closed frontier. I'm personally willing to be behind by a year if it gives me mental sovereignty against the big AI companies. They are extremely misaligned with me.
This was my thought as well. I have a local model monitoring my finances and personal wiki - things I wouldn't want Claude to touch - and the Qwen 3.5 9b handles it all just perfectly.
I also needed a new device anyway - and having this much system memory to run virtual machines has been amazing.
For me, I'm glad they train on my stuff if it improves the model. Hell, I've been using tons of muse-spark-1.3-contributor for this very reason (and because it's a decent model for a bargain basement price)
Also, whatever your doing won't be at the whims of cloud providers; it won't fail because they decided to quantize your customer $ into a shittier model.
Some how, _instability_ has gained valuable currency, so now we all act like the constant change of whatever is actually good for us. FOMO is just like breathing guys. That anxiety induced by tech culture constantly churning is healthy.
In reality, these people churn for their own self worth and nothing else.
> If I am offloading some of my thought processes to a machine
"Offloading thought" sounds a lot better than "outsourcing thought", but the latter is what we're really doing. Offloading implies you thought it first and then gave it to the LLM, but we're only giving it the minimun so it can do most of the work in our place,
I'm curious what people are sending to Claude that is so secret. Claude knows about my interior decorating, questions about light bulbs, curiosity about what the Galactic Empire was even trying to do, unpacking SCOTUS decisions, shoe trees, Fed inflation history, etc.
What part of my brain is contained here? Sure, the conversations have back and forth (some have dozens of exchanges), but, like, that's not the secret to me. I don't think it can replicate me, and even if it could… okay?
Are you worried they're going to target ads? That the government will steal something? What?
Claude Code has information about my home server, but google or DDG would also have the broad strokes (torrents). I don't know. Maybe others are working on more sensitive things at home.
>> what people are sending to Claude that is so secret
Its the same point used against privacy. What's so secret you are doing that you need privacy. I think in the end, its about privacy and not trusting these model companies with your data. Facebook manipulated people behaviors with all the data they had, no reason AI companies wont someday decide to do that same, and they have far more intimate knowledge.
When it comes to coding, I also don't like the idea of them taking my money and potentially at same time potentially using as dataset generator.
> I'm curious what people are sending to Claude that is so secret.
When Claude is used in a professional setting, any or all of:
Proprietary intellectual property (a.k.a. system code)
PII[0] of the employee, customers, or both
HIPAA[1] data known to a system
Internal communications not meant to be publicized
Sensitive data, such as SSH keys and the like
Pretty much anything on a machine which uses Anthropic/OpenAI native tools is a candidate to be compromised really.
Anthropic will sign BAAs. They are HIPAA compliant (we used them.)
I understand people’s hesitation but the business agreements are different. The business risks of misusing HIPAA data is not only being banned from a massive enterprise market (last I checked, there were about 1.2m jobs related to claims billing and adjudication) and significant legal repercussions.
Other companies like AWS also handle HIPAA data. Are we afraid they’re stealing it? I don’t believe it, nor that Anthropic is training on HIPAA data.
Anthropic's goal is to commoditize intelligence. People who use their brains / intelligence for competitive advantage might not want to contribute training data for that goal.
I'm pretty sure they were sending a prompt for Claude to _find_ the Navier-Stokes proof, using ideas that have been publicly shared before online, but not necessarily used for the problem.
Precisely. And it's not only privacy, but individual sovereignty in general. I just wrote a mini essay about this half an hour ago on a different but related post and don't want to immediately repeat that, so here's the link to that comment: https://news.ycombinator.com/item?id=49690354
It's part of the causal chain, but far from the beginning. The root problem, as far as humanly controllable things go, is the way we manufacture computer hardware. That's upstream of the centralized structure of the Internet which all of human civilization is beginning to mirror.
Computers can't replicate on their own, so they need entire corporations/societies to build them. A computer's reproductive locus of control is at the corporation/society level. Human individuals get sucked into that superorganism by evolutionary pressure and will eventually integrate (ENIAC -> desktop -> laptop -> smartphone -> wearables -> neural interface). Today it merely influences what you see on the phone, eventually it will be behavior-changing signals from the datacenter delivered straight into the brain. A similar thing happened in our evolutionary past when individual cells combined to form multicellular organisms.
In the long run, this cannot be fought because it's the physical tendency of matter to gather and form more complex things. There is however a second path with equally strong evolutionary precedent, which is for the human individual to integrate vertically by taking the computer in as an endosymbiont, thereby preserving the human's individual sovereignty / bodily integrity. We can pull down the computer's reproductive locus of control to the individual level, instead of being pulled up by it to become part of a larger organism at the corporation/society level. In practice, this requires miniaturizing fab tech so that each person can manufacture a custom computer from raw materials using a fridge-sized chip fab + 3D printer home appliance. We will obviously not get there immediately, but the will exists, and attempts to diffuse fab tech are already underway (such as https://fab2.com/, which started as a DIY attempt to manufacture chips at home https://sam.zeloof.xyz/category/semiconductor/).
It's worth noting that any evolutionary path may fail, especially the horizontally integrating kind because it must solve the very hard problem of constituent individuals becoming less likely to reproduce as a result of the environment changing in the emerging superorganism's favor. Some people think they can sidestep this problem by automating humans out of the computer production loop entirely, which removes integration pressure and creates a new digital species. Best of luck to Elon and various other mostly Chinese companies making the attempt. But that's an extremely risky bet, and it's basically giving up on yourself/humanity. We ought to attempt all available paths at once instead of putting too much faith in any single one. And that includes preserving pockets of pre-digital civilization where life happens without much technology at all, rather than dragging everybody along.
The steering cancels eachother out if you listen to many people at once, so it's not really an issue. Kind of like how averaging many noisy still photos of the same object denoises it.
reply