This has probably been posted before. As interest in hardware acceleration for AI intensifies, I started to think about whether analog circuits would make more sense. You can imagine actually modeling neural network architecture in the circuits and using circuit response functions as the activations. In theory you could make this super fast.
I'm sure people smarter and more knowledgeable than me have thought this through...
Yes. Each industry has developed accounting standards that reflect the nature of their business. You couldn't run a bank or a payments company with a simple sales - COGS = gross profit model, it just wouldn't make sense.
In health insurance specifically, profitability is somewhat regulated and this gets at the accounting issue here. Insurance companies should maintain a medical loss ratio of 80-85% meaning that fraction of the premiums should be paid to providers. The remaining 15-20% is split between administrative costs and profit. Most of the article's forensic arguments around this are weak and circular and represent a misunderstanding of the accounting itself.
Woodworking taught me a lot about planning and design. As a young person, I was like the authors brother. I just wanted to do the thing, not draw a diagram and figure out how much wood I need, or build a fixture to mark the stair lines.
Woodworking (the more constructive, furniture-making kind), rewards a deliberate, controlled process and it savagely penalizes mistakes. Those lessons transfer well to other disciplines. I’d have been a much better student if I’d learned wood working in high school.
Woodworking was part of my first 3 years of high school, but it was mainly about learning safety and tool usage and not planning, estimating, selecting or purchasing timber.
These days I only want to go to the lumberyard once for a project. Learnt the hard way on my first project that you need to take the time to carefully select the timber - checking straightness, matching grain and also colour before I started. Major hassle and waste of time to have to go back to swap boards.
That's also a lesson about what people will sell you. First time I went to a lumberyard, I was (coincidentally) with a friend who did a lot of woodwork. I thought, well, I've just paid for a pack of wood, I'll get it. The worker there was completely happy with that. My friend stopped me, and inspected each piece.
Sure enough, several had cracks at the ends, knots in poor places, and other things that, had I bought it, would have caused me trouble.
I can be a naive person in that I assume good faith. I would never knowingly sell something poor quality to someone else. I had assumed because I was being sold it, it was okay.
They aren’t “knowingly selling you poor quality” as some sort of scam. They are selling you wood to the spec you asked for. If you want higher-grade wood, you either have to spend money getting lumber graded to a higher spec or spend the time going through piles of low-spec boards to find the good ones. Many engineered wood structures are designed to use “poor-quality” wood, and they prefer it because it’s cheaper than using less high-grade wood.
The thing is, other packs of the same wood to the same spec were better. We were able to sort through and get one graded/rated the same, but without problems.
I know about wood quality and I have deliberately bought higher and lower grade wood. But even so, quality varies greatly.
Yeah, that is correct. Sawmills often produce only one or two grades of wood and don’t do aggressive binning. That’s why the quality is so variable within the grade. There are also factors that affect the grade but don’t necessarily impact every application (eg warping and knots are sometimes ok), so the bins are coarse-grained.
Yes but what about AI? (Perhaps the most annoying words written in the last few years mostly on LinkedIn).
But actually in the years since this was written, I do think the world has shifted. Doing things on a computer used to be really hard. Even just installing a framework or getting >python to call the right python on windows. Then install Django and get Django to work with nginx etc. It was just a lot of thankless, frustrating work to get from zero to 1%.
Aside from AI, the tools and packages and culture of computing has gotten better. But AI means you just get all the trivial but difficult stuff for free. And I think a lot of people who would have given up now make it through to see something work and they’ll feel the thrill of building something. It’s just better and easier now.
I mean no. The LIBOR analogy is appropriate. Large, long-term egg supply contracts are fixed to an index and that index was manipulated. That's criminal conspiracy and price fixing, not just a liquid market.
That's notably different from say the current scrum for HBM where the demand truly came as a surprise and scarce supply gets bid up.
Micron's windfall is justified and natural as these things go. The egg windfall was manufactured and criminal.
LIBOR didn't triple the rate. I don't doubt that they screwed around at the margins but the extreme volatility in egg prices were predominantly caused by the underlying economic factors.
What do you mean? Did you read the article? There’s so much evidence showing that it was the opposite.
Their profits shot up 3x in 2023 and 5x in 2024. They had 70%-140% profit margins. They publicly said that the end of the flu was a risk for their profits. There’s plenty of messaging recording explicit price collusion.
How is that a natural supply shortage?
The underlying economic factor is simply that monopolies or cartels will always try to manipulate prices in their favor if they can.
The indictment has them talking about 2 cent price swings. Like I said, this is goosing the margins. It's not an industry wide thing that tripled prices.
Yeah but the smoking gun is that volumes didn’t really go down much. I would buy higher unit margins on a shrinking volume, but for total profits to materially increase that means prices went up without a significant impact to volume. That smells rotten.
The author admits that the logic of the language and the design of the parser are idiosyncratic. Even the solution the author likes is an extension of an existing hacky trap door. He could be more open-minded about the solutions the AI proposed and in fact, I think AI could potentially rearchitect this in a more structured, sustainable, and legible way.
Many developer criticism of AI coders could be easily directed at 95%+ of human developers. Much coding is monkey see, monkey do and keep trying until it does the things we want it to do. AI can certainly do that cheaper and faster and really this is why automated testing became such an important software discipline with or without AI.
Maybe fair. I think my point was the author emphasizes how strange the software is. The further you are from the training data, the less well a model will perform. I haven't looked at the project, but it seems like it could maybe be written more conventionally. Or maybe not! In which case AI is bad at creativity and thinking outside the training data and that's a genuine insight.
Which is so weird, right? Like what is IBM now and how does a research lab make sense with the rest of their business?
The money-making parts of IBM are: legacy software and hardware (declining), consulting (low margin, low leverage), enterprise software (mostly redhat, not really growing). It's hard to explain how IBM research is accretive to any of that.
Licensing is a substantial source of revenue, and their servers have very impressive (think Telum’s caching) innovations, even though they rely on third-parties for manufacturing the chips themselves.
They are also betting on quantum computing to become commercially relevant.
The hardware division has 80%+ margin and still makes the systems that process 75% of all financial transactions. Their processors for those systems are on par or better than any other, I don’t think that is a business at all. This cash cow is not going away any time soon and gives them the profits to make bets on the future of computing.
I don't know enough about their business to say, but I'm thrilled at even the idea that someone might actually value long-term success over quarterly earnings.
IBM at one time had nearly everything in modern tech under their roof like Xerox and squandered it. There is no comeback for them.
They were the second American chip company that said no to Steve Jobs when hinted about designing smaller better chip mobile devices the other two were Motorola and Intel Apple had to eventually do it themselves. Apple Silicon
Doubt it. Consulting firms run 30% gross margin and operating margins in The single digits or teens. I’d bet IBMs legacy portfolio is 40%+ operating margins.
Sometimes you might hear private partnerships quote gross margins in the 40s or 50s but that’s implicitly gross of partner labor and comp so it’s not apples to apples.
Um, did i just get mansplained compound interest by Paul f*ing Graham? I feel like this has been the subject of condescending advice since the beginning of time.
"But now you at least understand, from having done the math yourselves, that you don't have to cheat to become a billionaire. You've seen for yourselves that there are only two numbers in the calculation, the growth rate and how long it continues."
What could possibly be false in a two-parameter model of reality?
I mean, I don’t know how long the NVIDIA moats can hold. With this much money at stake, others will challenge their dominance especially in a market as diverse and fragmented as advanced semiconductors.
I think that NVIDIA’s moat is the US government. Remember our government’s efforts to prevent the use of Huawei cell infrastructure in Europe and around the world?
I am a long time fan of Dave Sacks and the All In podcast ‘besties’ but now that he is ‘AI czar’ for our government it is interesting what he does not talk about. For example on a recent podcast he was pumping up AI as a long term solution to US economic woes, but a week before that podcast, a well known study was released that showed that 95% of new LLM/AI corporate projects were fails. Another thing that he swept under the rug was the recent Stanford study that 80% of US startups are saving money using less expensive Chinese (and Mistral, and Google Gemma??) models. When the Stanford study was released, I watched All In material for a few weeks, expecting David Sack’s take on the study. Not a word from him.
Apologies for this off-topic rant but I am really concerned how my country is spending resources on AI infrastructure. I think this is a massive bubble, but I am not sure how catastrophic the bubble will be.
I think the answer to your question is ‘not for very long.’ I frequently have breakfast with a friend who is a retired math professor and he is an avid investor in the stock market. We talk a lot about how long the US stock market will keep increasing in value. We don’t know the answer about the stock market, but it is fun to talk about. We both want to start easing out of the stock market.
They are heavy into AI investing but will tell people AI startups are just toy apps (Chamath). That podcast is full of crooks. I’d be willing to give them a pass as bunch of old white guy techies that just love to talk about tech, but they are literally at the dinner table with Trump and Musk.
This country used to have congressional hearings on all kinds of matters from baseball to the Mafia. Tech collusion and insider knowledge is not getting investigated. The All-in podcast requires serious investigation, with question #1 being “how the fuck did you guys manage to influence the White House?”.
Other notes:
- Many of them are technically illiterate
- They will speak in business talk , you won’t find a hint of intimate technical knowledge
- The more you watch it, the more you realize that money absolutely buys a seat at the table:
It's not startup success rate, it's application of the technology at companies. Meaning that 95% of the time that AI is applied to a work problem, it fails to generate material value over existing methods.
Sacks has always been absolutely disingenuous and interested in pedaling his own interests over the interests of the common good. As a total Trump shill he talks out of both sides of his mouth at the same time & accuses the left of things that he has no problem with when he or his own party does it.
Anyone who's listened to him (even those who align with him politically) for an extended period of time can't help but to notice so obviously so self interested to the point of total hypocrisy—the examples of which are too many to begin to even wanting to enumerate. Like—take the Trump/Epstein stuff, or the Elon/Trump fallout—topics he would absolutely lose his sh*t over if these were characters on the left. I find it hard to believe anyone actually ever took him seriously. Branding myself as a fan of his would just be a completely self-humiliating insult to my intelligence and my conscience IMO.
> a week before that podcast, a well known study was released that showed that 95% of new LLM/AI corporate projects were fails.
I mean. I think some of us knew this. There's a lot of issues with AI, some psychological, some are risk adverse individuals who would love to save hours, weeks, months, maybe years of time with AI, but if AI screws up, its bad, really bad, legal hell bad, unless you have a model with a 100% success rate for the task, it wont be used in certain fields.
I think in the more creative fields its very useful, since hallucinations are okay, its when you try to get realistic / look reasonably realistic (in the case of cartoons) that it gets iffy. Even so though, who wants to pay the true cost of AI? There's a big uphill cost involved.
It reminds me a lot of crypto mining, mostly because you need an insane amount to invest into before you become profitable.
I was also wondering if Google would try to make profit from selling TPUs, but they probably won’t because:
At least for me, Google has some real cachet and deserves kudos for not losing money selling Gemini services, at least I think it is plausible that they are already profitable, or soon will be. In the US, I get the impression that everyone else is burning money to get market share, but if I am wrong I would enjoy seeing evidence to the contrary. I suspect that Microsoft might be doing OK because of selling access to their infrastructure (just like Google).
There's no point selling TPUs when you can bundle TPU access as part of much more profitable training services. The margins are much higher providing a service as part of GCP versus selling.
I agree. Amazon and I think Microsoft are also working on their own NVIDIA replacement chips - it will be interesting to see if any companies start selling chips, or stick with services.
From what I'm hearing in my network, the name of the game is custom chips hyperoptimized for your own workloads.
A major reason Deepseek was so successful margins wise was because the team heavily understood Nvidia, CUDA, and Linux internals.
If you have an understanding of the intricacies of your custom ASIC's architecture, it's easier for you to solve perf issues, parallelize, and debug problems.
And then you can make up the cost by selling inference as a service.
> Amazon and I think Microsoft are also working on their own NVIDIA replacement chips
Not just them. I know of at least 4-5 other similar initiatives (some public like OpenAI's, another which is being contracted by a large nation, and a couple others which haven't been announced yet so I can't divulge).
Contract ASIC and GPU design is booming, and Broadcom, Marvell, HPE, Nvidia, and others are cashing in on it.
I wouldn't be surprised if a fair portion of Amazon's Bedrock traffic is being served by Inferentia silicon. Their margins on Anthropic models are razor thin and there's a lot of traffic, so there's definitely an incentive. Additionally, every model that's served by Inferentia frees up Nvidia capacity for either models that can't be so served or for selling to customers.
That's the same as saying that Google is winning the AI race because they don't like losing. They won't win anything if we are in a bubble that burst tho
A hypothetical AI bubble bursting doesn't mean that every single AI vendor fails completely. Like the Dot-Com Bubble, the market value drops precipitously and many companies fold, but because the market value does not fall to zero, the survivors (i.e. Amazon) still win.
Websites were still mostly selling goods and services in 2001. Not giving away hot takes and hallicinated summaries in exchange for eyeballs. In other words, after stuff like pets.com collapsed, people still found it useful to have pet food delivered, and the business model evolved. LLMs, on the other hand, don't seem to have a lot of public appeal. Most of the use cases are being shoved down the public's throat. Their appeal is to corporations as cost saving replacements for workers. But an AI bubble bursting would look like corporations rolling back their exuberance for the AI craze. What's already only speculatively profitable and requires enormous capex would probably become too toxic for anyone to try again for a generation.
As long as only TMSC is only top performance chip producer and it is possible to reserve all it manufacturing capacity for one two clients the NVIDIA will hold without problem...
My opinion, the problems for NVIDIA will start when China ramp up internal chip manufacturing performance enough to be in same order of magnitude as TMSC.
TSMC and NVIDIA's relationship has gone back for more than 20 years. In the NVIDIA biography they talk about how TSMC really helped NVIDIA out early on when other suppliers just couldn't meet the quality and rate demands that NVIDIA aspired to. That has led to a strong relationship where both sides have really helped each other out.
> But other people are literally getting their things fabbed by them.
Sure, but in my example Apple got access exclusively for a few months to a newer node, which would make a world of difference if you compete in the same space.
> but I wonder how important performance really is here.
Perf is important, but ime American MLEs are less likely to investigate GPU and OS internals to get maximum perf, and just throw money at the problem.
> solder on a large amount of high bandwidth memory and produce these cards relatively cheaply
HBM is somewhat limited in China as well. CXMT is around 3-4 years behind other HBM vendors.
That said, you don't need the latest and most performant GPUs if you can tune older GPUs and parallelize training at a large scale.
-----------
IMO, Model training is an embarrassingly parallel problem, and a large enough cluster leveraging 1-2 generation older architectures that is heavily tuned should be able to provide similar performance to train models.
This is why I bemoan America's failures at OS internals and systems education. You have entire generations of "ML Engineers" and researchers in the US who don't know their way around CUDA or Infiniband optimization or the ins-and-outs of the Linux kernel.
They're just boffins who like math and using wrappers.
That said, I'd be cautious to trust a press release or secondhand report from CCTV, especially after the Kirin 9000 saga and SMIC.
But arguably, it doesn't matter - even if Alibaba's system isn't comparably performant to an H20, if it can be manufactured at scale without eating Nvidia's margins, it's good enough.
> Their multiples don't seem sustainable so they are likely to fall at some point but when is tricky.
They've been trying really hard to pivot and find new growth areas. They've taken their "inflated" stock price as capital to invest in many other companies. If at least some of these bets pay off it's not so bad.
Chinese semiconductor dominance is not imminent and US containment has been somewhat effective. I don’t think that will hold on a generational timeline, but it will be hard to overcome.
You don’t think the export controls on Nvidia chips accelerated Chinese investment in ML processors and therefore their independence -> dominance in the space?
The export controls made it difficult for Chinese companies to acquire large numbers of GPUs, which prevented them from expanding business models that rely on buying more GPUs to serve more customers, which means that Chinese companies have much, much lower budgets for GPU procurement than their American counterparts. https://chinai.substack.com/p/chinai-323-the-ai-deflation-of...
So a new homegrown chip would have to capture a very large share of this relatively small market to make significant volume. That makes it rather risky for profit-driven investors.
Politically directed investment probably increased, but in the end the private sector also needs to be on board.
Most of the rest of the world can buy Nvidia if they want to. And I don't think the EV comparison works, since China already had a large domestic market and established ICE car companies who could afford to electrify some of their lineup and slowly gain market share this way.
That is not at all the case. Biden export controls for nvida chips are still in place. China also has a large domestic market for chips. The rest of the world would gladly buy competitive Chinese chips.
Building SOTA semiconductors is more art than science. All the best artists are in Taiwan.
You don't just buy (or copy) and ASML litho and turn it on. Just like you don't buy a horsehair brush and start painting Picassos. It's even more difficult than that because there are something like ~1000 sub processes and each one needs a world renowned artist in that specific art to get it done.
Its the reason why even Samsung can't match TSMC despite having the same tools and capital.
> So will the military hopes of ever protecting Taiwan.
I don't think there are too many military analysts who would claim that the USA could "protect" Taiwan if China was really determined. The USA still retains the ability to significantly increase the cost, both militarily and economically, of an invasion, and relies on this - successfully, to date - as a deterrent.
I think most people recognize however, even in Taiwan, that in terms of pure practical facts on the ground, not even the world-beating US military can overcome geography. Taiwan is 100 miles off China's cost, Guam is 2800 miles away. It is difficult for me to imagine anything, barring some incredible technological advantage that the USA shows no sign of possessing, that could outweigh such a tremendous home-turf advantage.
It is very hard to come up with a realistic, or even semi realistic, scenario in which China does not end up with Taiwan if it really wanted it.
That's a very pessimistic take, or optimistic I guess, depending on perspective.
Looking at the Chinese semiconductor development trajectory, and considering that TSMC won't be sitting on their hands, "within a decade" seems really unlikely.
I am aware, I live in Taiwan. While TSMC engineers can be poached "move between countries freely" is not true because moving from China to Taiwan is not so easy.
The key word you mention is "replicating." They'll be chasing for a while still, and it's not clear that they'll be able to leap ahead. Copying is much easier than real innovation.
I was under the impression, for years, that the US had the appropriate government, scientists and engineering in place to protect the castle. However given what I've seen in the last few years - I agree that it seems inevitable China will surpass the US in the next decade and will hold both cards and a grudge.
It's amazing how China has doubled down into STEM and green energy while the US has done exactly the opposite. The CHIPS Act propped up a company further driven into the ground by Pat Gelsinger. The last few administrations have had no focus on driving innovation and technology - only propping up the Tech Bro market making money off of attention and ads. Maybe, just maybe, the US should stop electing geriatric and short term gains ignorance?
The US needs to dig its head out of its ass if it wants to continue to be recognized as the global power it once was.
Taiwan could be protected if they were given practical control of nuclear weapons or similar, i.e. nuclear weapons sharing.
Securing Taiwanese independence is going to be necessary for the EU to ensure that there isn't a US microchip monopoly, and the only way the EU can do this is by the aforementioned means.
Zelensky at the MSC 2022: “If … results do not guarantee security for our country, Ukraine will have every right to believe that the Budapest Memorandum is not working and all the package decisions of 1994 are in doubt”[0]. 4 days later Russia invaded Ukraine. At the time it was said that Ukraine pursuing nuclear weapons was the last straw, but who know really.
If Taiwan is going to place nuclear weapons on its territory the conflict would probably escalate quickly.
Could well be a trigger. This is a reason to do it quickly by having already existing, tested nukes handed over for use by the threatened state.
There is a problem with this kind of reasoning though. It's like, somebody shows up 'try to grab my gun and I'll shoot' but the thing is, he might shoot anyway, and if you grab the gun, maybe you'll get hold of it.
When someone threatens something, that's not reason for stepping down, passivity or anything like that, it's reason to immediately risk everything on an attack to take away the thing he has that allows him to threaten you. By threatening something he only demonstrates that he must be attacked immediately.
I like an example I gave earlier with hostages. It's Monday, someone has taken a hostage and threatens to kill unless left alone. The next day he's taken another, he has the same threat. On friday, he has five. Now, you realise you should have attacked on Monday when your attack only risked one death.
So today, the question might be 'why didn't we start building nukes last year?'
I'm sure people smarter and more knowledgeable than me have thought this through...