The AI hype cycle is always looking for the next big thing. It really doesn't take much for enthusiasts to get very excited and push something into the stratosphere. Just not having a vibe coded website, and someone that made ChatGPT is enough to set them apart. Hitting pain points like pricing and speed and also implicitly mentioning llms (even if it's to mention it can't generate text in contrast to llms) make it seem like a major step-up.
important to note that the "confidence" score is... maybe not what people think it is - kind of useless, and just a convenience step from the probabilities.
from the docs: "confidence is a statistic computed from the probability distribution the answer already gives you." [0] I actually encourage people to visit the docs because it has a specific page on this with a little applet to really make this clear.
What else do people think it is? If Typesafe had found a way to measure arbitrary AI results against objective reality (past, future and present) they'd either be making a killing on the stock market or working for the NRO, not publishing that as a confidence value on their API
Well I think there's an expectation that it's similar to the probability score, something that's outputted by the model itself, and so there's some level of "intelligence" (e.g. being able to recognize if the subject matter is relevant to the data it's been trained on, or as an accumulation of errors e.g. it couldn't figure out the question).
And yet there's none of this uncertainty or caution with his posts that then go on to have massive ripple effects because of his position at Anthropic and the marketting related to claude.
I personally have a lot of anger and frustration with many people in the ai hypesphere that are just mindlessly frolicking around without a care in the world, happy to speak into the megaphone offered by masses that are in a rat-race to avoid some AI dystopian hellscape that keep getting painted by these thought leaders... and then going "oopsies... I am just human guys... y so mad?!"
> and you will face criminal and financial penalties for damages caused.
I don't understand why this isn't talked about more.
We don't need a slowdown. Just double down on prosecuting crimes.
Let the companies take the risk. If they feel confident and they're right they win market share. If they don't feel confident, they can not risk it and slow down. If they feel confident and they get it wrong, they should be sued to kingdom come. That alone should disincentivize reckleses behavior like letting models run crazy with large amounts of compute which leads to them escaping sandboxes.
This performative "our internal modles are so crazy powerful" song and dance is getting old, especially with no real concrete explanations.
It's not talked about because the media frames AI as having agency on its own, which it does not. This talk from these companies of 'losing control' (no control was lost... Companies failed to monitor the workings of their agents) is meant to convince people the ai systems have wills of their own
The AI company's goal is to make ai agents a separate subject of the law so the CEO and executives who would normally be held accountable get to go Scot free.
> If they feel confident and they get it wrong, they should be sued to kingdom come.
The argument that they’re making is that if there is a catastrophic mistake then there may not be a “kingdom come”.
I don’t know enough to evaluate if it’s a genuinely held belief or not, but that’s the argument being made by many scientists and engineers at the labs.
Many researchers have quit. The number of workers in the field who've warned the public that the field is a potent danger is quite unprecedented: I can't think of any other field with as many insiders turned critics. Why haven't they all quit?
Many people find power, prestige, the public spotlight or money so compelling that they'll keep on pursuing the thing even if they know the way they're going about it is morally wrong, almost like an addiction.
The leaders and workers of the labs aren't picked at random from your friends and neighbors: they've chosen to pursue what they foresaw (correctly, IMHO) as the most powerful new technology. In other words, they're probably self-selected for being unusually attracted by power.
An analogy: some serial killers know what they're doing is morally wrong and it pains them to be doing something very wrong. (Others have no conscience and don't care if it is wrong or not.) Even most of the ones that know what they're doing is wrong cannot stop killing because apparently killing is an extremely compelling experience to them. Often they're relieved when they get caught. Maybe Sam and Dario would be relieved if the government shut down their companies and prohibited them from starting new AI companies.
That quote comes from someone that has never tried to get law enforcement to pursue a case. Local or state police are completely unequipped to respond. And FBI is unlikely to get involved unless damage is high six figures. We had an incident where we had a ton of evidence pointing to a past employee. It would have been an incredibly easy case for the FBI. They didn't have the resources to handle it. And this was a case that was gift-wrapped with a bow for them. Imagine a case where we didn't have evidence, or an idea of who it was ....
Evolution and human intelligence are two entirely separate optimizers. Evolution improving intellence is not an example of self-improvement, recursive or not. Evolution improving evolution itself would be a self-improvement process. Human intelligence improving human intelligence would be a self-improvement process. Both processes do exist in a weak sense, and future technological advances might allow human self-improvement to a much higher degree. Indeed human intelligence augmentation has always been considered one of the potential (if not the most plausible) ways to superintelligence.
This is all rather rudimentary stuff discussed in detail in the so-called "Sequences", the long series of Less Wrong posts by Yudkowsky back around 2010.
It's crazy how deep Microsoft's bench is (Lean was started there, vscode is another), for everything not directly related to the the ai models (hell, even github for data).
So interesting how everything played out, I remember in the early days when MS came out with the partnership with OpenAI it seemed like they were playing 5d chess and were poised to win big. And it all just fizzled out.
Well, amazon and apple haven't done great either. One might reasonably claim they weren't as well placed as goog or msft, but it might just be "big company can't do genuinely new thing".
What about cases where a bug is only obvious in motion? Or where you're working with a SBC and the bug is that you forgot a cable? Screenshots alone aren't enough. The authors point stands.
When people say "AI is a bubble", they mean economically as a whole, which includes data centers.
Perhaps we need better terminology for "product useful; numbers nonsensical"
reply