"Baking in" a model into a chip is a bad idea because chips take 2 years to tape out and then you're stuck doing inference on llama 3 in 2026 when fable/sol are available. Every accelerator is a tradeoff between flexibility and performance and GPUs are already pareto-optimal
There are other options. I worked for a startup called NVXL and we were programming DNNs into FPGAs using OpenCL, on custom boards we built to plug into NVME. It worked great, but it wasn't fast enough at the time to compete with Nvidia, or even Intel AVX512. Ultimately the company failed, but maybe some hybrid like that could work for LLMs? I haven't been up to date on how DNNs and LLMs look like under the hood these days, but there's got to be someone doing something similar.
That's only half the problem. OpenAI is contractually obligated, if you will, to believe that models will continue improving at an impressive rate for the foreseeable future (otherwise their valuation makes no sense).
If you believe that, then you should expect to get Sol-level performance out of a Luna-cost model within six months or a year. If you have a system with the weights baked in, that means you're going to end up serving that Sol-class model several times more expensively than it will take someone who comes along a few months later. (such as what recently happened with DeepSeek's update.)
And under that assumption of continuing advancement, baking things in doesn't make sense in general - it's a play you'd make if you think things are slowing down a lot. Which may be right but it's not OpenAI or anthropic's play.
Assume a $800 billion valuation. $100 billion ad network. $30 billion op income. 26x price to op income ratio. It's right there for them to grab, or someone else to grab.
Their valuation does make sense if you believe: 1) they can retain a massive user base and 2) a massive user base can be monetized. Future value is almost always pulled forward these days for high growth tech companies.
An LLM the size of Google search in users is even more valuable than Google search. The ad market for LLMs will be even larger than search was (no matter what HN prefers).
The monetization part is the easier part. Silicon Valley understands extraordinarily well how to build ad networks. If OpenAI maintain their gigantic user base, a $100 billion ad network is a given bolt-on. They'd have to screw that up in an epic way to not get there.
Facebook - Insta - WhatsApp is an absolute dogshit tandem with a gigantic user base. $228 billion in ad sales and still expanding 10% per year.
Google knows this is what's happening, that's why they don't care about chasing Anthropic very much. They're busy completely remaking how their core search business works.
If a strange and quirky architecture can increase their margins, they could start becoming so wildly profitable that they don't care about their investor driven valuation anymore.
The whole point is that it's supposed to be more efficient. But models are also still getting absurdly more efficient every year, so you're likely nullifying much/most of the advantage. 18 months is a long time right now (and 18 is only time to tape out, not operational in data centers).
Even if the balance was net positive, you would also not be able to train them against new tools/harnesses or knowledge. How many years do you expect to keep using them?
Good enough will hit when the tech stops advancing quickly. You could have a "good enough" model but in 2 years if the general purpose chip can run it just as fast, there is no point having the single purpose one.
The good enough level isn't ever arriving. We're in the first or second inning for LLMs. They will rapidly subdivide in complexity, they will not stagnate in the next decade.
Beyond the model, when would you freeze processor performance, such that it was good enough? Because that's exactly what freezing on Talaas is premised around.
The semiconductor technology will also continue to improve. You lose twice. Talaas is one of the dumbest ideas I've seen in semiconductors in decades.
Closer to 2 weeks, as long as the new model fits: your model bits are entirely in mask rom, so you can change them with a metal only ECO that only touches two metal layers. You probably don't even need to run any timing analysis, etc. since all of the changes are going to be isolated in a gigantic square that's isolated from everything else.
I think it would make sense to upload the weights like a firmware into the chip rather than baking the weights itself directly onto the chip. If there is an option of easily updating the firmware from time to time it would work well.
I've seen several attempts even on HN of the LLM meets Teddy Ruxpin (or more accurately AG Talking Bear) but most of them offloaded the AI to some off-site servers.
I’d like to think that most parents would be weary of handing their children what basically amounts to a tape recorder that siphons all the data off to a large corporation.
OTOH, a completely local one (LLM + VAD + Speech Rec) would be a fun little thing to build.
This is par for the course with exercise science. It's mostly fake. No blinding, small sample sizes, researchers with agenda, low duration, low funding etc. The good news is that doing almost anything works.
You're right, it's impossible to blind subjects. Researchers can be blinded by having one limb be the control and the other be the test. This design has become more popular recently and it's definitely a small improvement.
Progress, over time, tends to involve both variation in routine and specific methods, progression, programming, modalities, techniques, form, movements, etc.
One somewhat dubious 10 week study of newbies, as many others have commented, doesn't communicate much.
A further complication is that much of the hypertrophic adaptation is systemic, that is, relates to overall body stimulus and other factors (nutrition, rest, genetics, etc.). Among those effects is the net hormonal response (testosterone, HGH, ILG
Heck, there's a well-known phenomenon called cross education* where an untrained limb will see strength / hypertrophy gains when its opposite is trained:
Pretty clear trend: low-trust societies have low gdp and high-trust societies have high gdp, regardless of resource distribution. Africa/South America are resource rich, japan/iceland are resource poor.
Which direction does the causation within that correlation (if any) flow?
I can easily conjure a scenario where high per-capita GDP makes trusting easier (either because there’s enough to go around and/or because there are reliable police/judicial sanctions for violating trust) than in a hardscrabble low per-capita GDP society with lower (insufficient?) lawfulness.
This is just totally untrue, Epic is fine as a Dev. I have many friends who work there, none of them work more than 40 hours a week and they all make ~200k with 5 yoe. Great for Madison WI. Tech stack does suck, however.
Glad your friends' experiences are exceptional. I know a woman working for Epic in the Madison area who miscarried due to work stress. Madison housing is also a lot less affordable these days, assuming you want to actually live in Madison and not a copy-paste Verona suburb w/ zero walkability:
I know locals like to complain about the beltline, but as someone who has driven in LA, SF, Chicago, NY, it is wild to compare it to any big city traffic.
It's not in the same league as LA, NY, or Chicago but definitely up there (albeit during shorter time windows) with Silicon Valley commuter traffic. Part of a common theme of living in Madison where you get downsides of city living- traffic, high taxes, high housing costs- without many upsides such as good public transport, a diversified job market, or a well-developed food scene.
Those special cases are useful or the natural result of speakers coming from other languages. If esperanto were used widely for many years, it would also develop special cases over time. In 200 years, it would be as irregular as english, as it borrowed words and phrases from other languages.
those idioms and slang still follow grammar rules though. they are not irregular.
also the irregularities in english developed over centuries by contact with other languages in an environment that lacked the normative influence of globalization and modern communication tools.
at the time english was also not the defacto language for international communication that it is today. this status is responsible for efforts such as a wikipedia in basic english that makes information easier to approach by non-native speakers.
as esperanto was expressly created with the goal of being an international language, any attempts to introduce irregularities will meet a lot more resistance and therefore development of an irregular grammar is unlikely to happen.
In the StableHLO spec, we are talking about this in more abstract terms - "StableHLO opset" - to be able to unambiguously reason about the semantics of StableHLO programs. However, in practice the StableHLO dialect is the primary implementation of the opset at the moment.
I wrote "primary implementation" because e.g. there is also ongoing work on adding StableHLO support to the TFLite flatbuffer schema: https://github.com/tensorflow/tensorflow/blob/master/tensorf.... Having an abstract notion of the StableHLO opset enables us to have a source of truth that all the implementations correspond to.
It's conspiratorial to assume that advertising and marketing is responsible for the status quo, rather than it being reflective of the genuine preferences of consumers to have new toys. Consumer preference has rewarded lowest-common-denominator dopamine trains in basically every facet of modern life. See the gaming industry with microtransactions/lootboxes, or entertainment with the move towards short-form content.
Marketing does influence behavior. However, reward loops don't need marketing, just like heroin doesn't need marketing. I've sank 100s of hours into idle games that look like a spreadsheet and have 0 marketing budget.
It’s not conspirational to assume that advertising and marketing is part of forming the status quo, by virtue of those fields literally exploiting human weaknesses to do so. If they simply reflected what consumer want, not only would they not have a reason to exist, but if they did they wouldn’t need to resort to endless amounts of FOMO to get you to buy Season 329 of their Battle Pass.
In 2022, it would be stupid to make a game that doesn't have some sort of battle pass and dopamine-drip progression system. Way fewer people would play your game and gamers would complain without them. People genuinely enjoy these systems, just like they genuinely enjoy reality TV over a documentary or TikTok over long-form Youtube videos. It's an obvious trend that won't be fixed if we misunderstand the issue and assume that people don't want these things and they're being pushed by advertising.
How do you know people want these things? Do you actually think “yay! I love that this new game has a battle pass system and lootboxes” is a common sentiment? The point is that there is no choice. Sure, companies are very likely making more money off these systems than otherwise, but to say this is because people want it doesn’t ring true to me, simply because they can’t just pivot to a similar game without them, or avoid it entirely. Games like Fortnite are so predatory on monetization that they strip you out of even choosing the base nothingburger characters available for free, instead randomizing which one you get per match to force you to buy something to even have a consistent player model you somewhat like. That kids, a demographic that’s by nature desperate to express themselves, spend piles of money on these systems is no surprise. Not to mention the fact that they’re too young to know there’s even an alternative at all.
There’s also mountains of evidence that contradict the statement that a game without a battle pass won’t succeed. One of the most profitable and respected MMOs currently is Final Fantasy XIV, a game that has nothing like it, in the genre that had micro-transactions and lootboxes before most people even heard the terms. Middle Earth Shadows of War is also a great game that was ruined by loot boxes and progression systems, where the outcry was so big that the developers completely removed them after the fact.
Two economists walk by an exotic car dealership. One economist points at a Ferrari and says to the other, "I want one of those." The other economist says, "Obviously you don't, otherwise you'd have one already!"
Sugar is bad, but it's only one part of the picture:
"Evidence-informed dietary priorities include increased fruits, nonstarchy vegetables, nuts, legumes, fish, vegetable oils, yogurt, and minimally processed whole grains; and fewer red meats, processed (e.g., sodium-preserved) meats, and foods rich in refined grains, starch, added sugars, salt, and trans fat."
Thanks friend!
For both lifts, I work up to a top single around RPE 8 (difficult, but with no grinding or form breakdown), and then use that number to determine my 1 rep max for that particular day. Then I do volume based on the Prilepin chart. I find this works very well because it adapts to the stress of life.
