You can pay for higher cache time, you can pay for NVMe KV cache for an hour that can just be reloaded, etc., at a lesser tier you can pay for the KV cache to be stored on a network store (I guess I'm unclear if that last tier would be cheaper than recomputation, not even 100% sure of the NVMe with direct GPU<->storage DMA) depending on your model settings.
It's entirely possible a game manages paging VRAM badly. But allowing the game more flexibility isn't a problem with a larger BAR, it's that the game is stupid and gets dumber the more VRAM you give it.
If a program runs slower when you give it more RAM, the problem isn't giving more RAM.
They’ll try to claim a non religious workplace code, an extension of the current apolitical code. Leave your politics at home becomes leave your politics and religion at home.
If it's closed source, how do you know it's not technology given by aliens from the 43rd dimension running on quantum computers enabled by discovering that P=NP and finding a linear time reduction from NP to P?
Occam's razor is that it's probably not all that different unless there is some specific reason to believe otherwise.
That goes all the way back to at least to Stein's Paradox in 1955, sadly too few people get educated about Statistics and keep thinking specialized models will necessarily be better. If you want to estimate the batting averages of 3 MLB baseball players from samples, you are better off building a model to predict all of their batting averages than computing the mean from a sample of each one separately.
You are saying something interesting, but talking like Grok and skipping a lot of the details, without any references to common check-in points like terminology or specific studies.
> and concentrate the likelihood around the zero loss set. Then reduce the variance on a Gaussian prior.
Those phrases could mean a lot of different things. What are you proposing?
> so that any measure of model quality will monotonically increase with model size and achieve a maximum at infinite model size.
any measure of model quality? You must have some bounds of any measure, since trivially that's false because "fewer parameters is better" is a measure of model quality, even if dumb.
It's hard to even engage when you're being so imprecise, and not even giving one specific example.
Apologies, I'm skipping details, because that's how I speak with my colleagues, but I realize this is an external environment without context. No references since this is folklore (you can look at Hastie et al's Surprises in High-Dimensional Ridgeless Regression paper for the non-Bayesian version, Bruno Loureiro or Andrew Gordon Wilson probably have a paper with something similar).
Concentrating a density around a zero set means that I raise it to the power of 1/gamma (appropriately normalizing) and then take gamma to zero. If the likelihood was Gaussian, this would be equivalent to taking the variance to zero (yielding a point mass). But in overparameterized settings, this concentrates on a submanifold describing the set of interpolating solutions. In least-squares linear regression, that is the solution space. Reducing the variance on a Gaussian prior is treated as an asymptotic expansion by Laplace's method. If you choose the variance to decrease (inversely proportional to the parameter size, for example), then the marginal likelihood will increase monotonically with model size.
By any measure of model size, I mean that you can pick your favourite among the common ones, such as information metrics (e.g. mutual information / KL), statistical metrics (e.g. marginal likelihood), test error. You should be able to show the same phenomenon happening for all of them, so it isn't a quirk of marginal likelihood. It is concentration of measure working in your favor to reduce the variance in the estimator.
No, I am talking about out of sample error and estimates thereof. It is "overfitting" to data, but it also has lower out of sample error than the case where you do not "overfit".
This is why the notion of overfitting is not nearly as cut and dry as a basic ML course would have you believe. Just because you fit data exactly does not mean that your estimator has high error on out of sample data. A trivial counterexample is a spiking model that spikes to fit to the data but otherwise follows the correct trend outside of the dataset. The bias variance tradeoff gets thrown out at enormous scale and overfitting is not a meaningful concept. What matters is regularization and robustness, not how well you fit the data.
The reason why bias variance tradeoff and considerations of model size are a good approximation for smaller models is due to concentration of measure in the data which effectively kills any regularization in your modelling procedure. Once you enter settings where concentration of measure begins to bite in parameter space, everything changes. This isn't really that mysterious; any textbook on Gaussian processes (e.g. Rasmussen and Williams) will tell you this.
Huh. This reminds me of the asymptotic equipartition theorem. Samples taken from higher and higher dimensional spaces will concentrate into a typical set.
Does model performance also concentrate into a 'typical case' where things work pretty well and a non-typical case where it's completely unpredictable as the number of parameters increase?
I don't understand what you mean. Test error is literally out of sample error. Marginal likelihood is designed to estimate out of sample error. The whole discussion is about out of sample; nothing has been about in-sample error. The in-sample error for my examples are all trivially zero, so only out of sample error is worth discussing.
> because it means subtle changes (introducing a destructor, re-ordering code, etc) can change semantics without you realizing it.
No, it won't change semantics - if you say @musttail or similar, it will simply fail to compile if you, say, introduce a destructor - the semantics will not subtly change.
Uh, yes, if you guarantee the semantics only when the code explicitly opts in and not by default then semantics will not subtly change, that is the point of my comment
Guaranteeing an optimization that otherwise only might run is a change in semantics. The attribute doesn't allow (in any sensible language) the code to simply not compile because the optimizer doesn't feel like it today (or you compiled with -O0), it forces the compiler to not allocate a stack frame wherever the code fits the structure that makes that definitely possible and fails to compile wherever it doesn't (even if after other optimization passes it happens to fit a structure that makes it possible).
AWS and Azure give you the same thing for Claude and ChatGPT, no need to be stuck with open weights. They might sometimes store some of it for other purposes (I don't know the specifics), but it is emphatically not being fed back to OpenAI or Anthropic.
Seems naive to think that those providers - who have a financial interest in selling the data - would not also try to weasel out of the precise definition of ‘zero’ retention.
You can pay for higher cache time, you can pay for NVMe KV cache for an hour that can just be reloaded, etc., at a lesser tier you can pay for the KV cache to be stored on a network store (I guess I'm unclear if that last tier would be cheaper than recomputation, not even 100% sure of the NVMe with direct GPU<->storage DMA) depending on your model settings.
reply