Self hosted database access management, think Code Reviews for SQL Statements for when your devs need to go to prod:
https://github.com/kviklet/kviklet
Recently cleaned up my postgres proxy and built a MySQL wire proxy so that you can use psql/datagrip as a dev but every statement is still logged.
With SSO and no password sharing ofc!
> measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer.
This is so useful because it makes you actually trust a models output. A high score on benchmarks is not as useful because a model overtrained to always answer will give confidently wrong responses. But this index measures how often it is correct while penalizing wrong responses so that a high score means you can trust this model more and when it doesn't know it is more likely to tell you that it really doesn't know rather than making shit up.
Fable also performs a lot better than opus 5 here which correlates very strongly with perceived strength despite the models performing similarly on e.g. DeepSWE
Astra is a big jump from sol and performs the same or slightly better than fable here.
Hallucinations are very damaging to a model’s utility. But doesn’t the Omniscience Index focus on knowledge-based queries? To me, using LLMs for their memorized knowledge is very 2023 and suboptimal.
IMO, what really makes a model useful is its ability to process information within its context reliably and faithfully. I don’t care if it hallucinates George Washington’s favorite color, but I do care about it hallucinating the results of tool calls.
Fair but I think those two behaviours are strongly correlated at least the index does represent my personal experience very well where fable is way better than opus opus is better than sol. And I haven't tried Astra yet but it having 44/43 is very interesting at least.
I don't agree. The main issue with their scoring/methodology is that the numbers make it seem like 5-6 models have little to no difference when in fact there is a significant difference between fable and opus and sol and astra for example. They are popular mainstream but most of their benchmarks are either not a representation of model strengths enough or they are not doing a good job of showcasing it properly. The fact that muse and 3.8 were high a day back shows they are just the modern version of lmareana for the mass audience and PR stunts.
The Epoch Capabilities Index uses an Elo-based aggregation method that dynamically adjusts for benchmark difficulty and they put error bars on their scores, both of which put them miles ahead of Artificial Analysis: https://epoch.ai/eci?view=graph&tab=leaderboard
> A high score on benchmarks is not as useful because a model overtrained to always answer will give confidently wrong responses.
It's not useful because the benchmarks often measure the wrong thing. They're here yapping about AGI and yet the benchmarks treat it like a trained dog. Fetch this. 100 points.
Each "problem" in these benchmarks likely has more than 1 solution that can be considered correct and even should be graded in many ways. Yet we see in many benchmarks higher effort (or thinking levels) don't help because the benchmark penalizes for doing "more" than what the answers asks for. So what did you ask for?
In human school you often get marks on the process and not just the end result. Thinking tokens have been cut. All we group on is things like cost, turns and time but not the what else.
This matches what I've seen building anything that uses an LLM for narrow structured output rather than open ended chat, things like classifying a diff into a fixed set of categories or summarizing a change. The aggregate benchmark score barely predicts how it behaves in production. What actually breaks a pipeline is a confident wrong answer on the small slice of inputs that don't fit the pattern it saw during training, not a lack of raw capability. A model that says it isn't sure on the edge cases is far more useful to me than one that scores higher on average but never admits uncertainty, because the wrong but confident output is the one that slips through review unnoticed.
It's really funny (and a bit obnoxious) to watch the vocabulary from the outside. In 2023 everybody learned the word moat, then it's been agent(ic), from last year there's more talking about harnesses than at a bungee jumping convention. The mot du jour is frontier.
It truly proves like there's a handful of thought leaders on Twitter that everybody follows blindly and start to copy down to the lexicon and parrot everywhere else.
I find the vocabulary used to be disturbingly fascinating, to the degree concepts are being anthropomorphized. It's so pervasive that I cannot help but assume it is entirely deliberate.
The principles by which LLMs functions haven't changed in the last four years. It is still a next-word-predictor, a statistical parrot, if you will. But if you don't understand the mechanisms behind it, you cannot be faulted for thinking this is something much more. Most of it, is pretty devious marketing.
As an example, no LLM model does anything that can be considered "reasoning", or "intelligent" in the traditional sense, but these words are used extensively. "High reasoning model" is a pricing tier. The article in question has an anthropomorphized term in every single sentence. I'll pick a paragraph at random and highlight the cases. If both parties understand the mechanisms, these words are fine, and we do that all that time. The issue is when one sides is mislead to believe that these systems can be relied on in a way that they should not be, leading to people getting hurt.
> > The *translation layer* is what *allows* a *harness* to *work* with different AI models. In some cases, a *harness may decide* to *use* different models within the same *agentic loop*, because different AI *models may excel* at different tasks. The *translation layer* is also a crucial aspect of *harnesses* because they *deliver control* to the end user. It means that someone can take their *AI harness* and use it with a model from Anthropic, or OpenAI, or explore one of the open weight AI models that often deliver great value-for-money (measured by cost-per-task).
The underlying logic isn't remotely as mysterious or mystic as the language makes it seem. A different paragraph:
> > Tools are a set of *capabilities*, written in code, that the model can *“call”*. The *harness describes* the tools and also *provides* the software that is the tool itself. Examples of these tools might include a web search tool, a tool that *allows the model* to write and execute software code, or a tool that *allows the model* to *compose* an email. Critically, the *harness usually* does not *dictate* when and how the *AI* model should *use* the tool. Instead, it simply *makes* the tools available, *describes* them clearly, and *allows* the *AI* model itself to *decide* when and how *it should use* them
My guess is that everything "reliable" in LLM/agentic-coding comes down to either calls to reliable/deterministic tools or providing well-defined success criteria (such as loads of unit tests) for the LLM to throw its stuff at in "agentic loops" until something sticks.
Instagram, Facebook and even threads all had much more mundane growth rates and definitely no unexpected jumps like GitHub is experiencing. I'm sure if suddenly the solar system had 10 more earths with each about 10 billion people and they would all start using Instagram tomorrow we would have exactly the same growing pains and outages that GitHub has today.
Luckily for Meta agents are not yet as much into doomscrolling as humans are.
1 in 3 is not terrible you just need a few more humans in the loop to reduce the error rate meaningfully. Combined with other classifier models and heuristics you can get good results. Humans can probably also perform better if they don't have to judge every single command but just suspicious ones our attention is limited after all.
1 in 3 is end of the line awful... Back when I was in college (former USSR), we had a subject roughly translated as "integration with industrial processes". USSR industry was highly regimented. Various norms, tolerances, recipes etc. were described in GOSTs (a kind of arsenal of industry standards). There were also some common knowledge / statistical bits that went into making these GOSTs. I mention this because this system dealt in great detail with quantifying human error (as well as errors resulting from equipment use etc.).
One of the core assumptions was that outside of extraordinary circumstances, the expected rate of human error is about 5%. However, the course also provided examples where error rates were significantly lower (eg. nurses in maternity wards would have a much lower than 0.1% error rate when pairing mothers with newborns).
The error rates, of course, also depended on human ability to measure the difference. Since I was studying typography, the printing process was of particular interest. A GOST for offset printing required that the color intensity for each ink of CMYK, for example, should be within +-2.5% range of the intended intensity. This is difficult for someone who doesn't have a lot of experience operating an offset printing machine to spot, but experienced printers have no problem with that.
Most importantly. There was never an acceptable error rate of 33%. Not for anything. If people were likely to make that many errors (eg. because the measurement was too difficult), that product would never have been allowed into production.
1/3, but under unreasonable time pressure, and with no prior vetting. For example, I played a few times, and I'm not a JS developer. I had to just suss out if npm whatever is dangerous or not. I'm very happy with my personal 25%.
The study he cites is also specifically using digital pens
> Brain electrical activity was recorded in 36 university students as they were handwriting visually presented words using a digital pen and typewriting the words on a keyboard
kinda funny.
It's also not clear from this study that there is actually any benefit to "more brain activity". Of course doing more complex motor tasks requires more brain activity but nobody guarantees that this helps with remembering things.
i had no science, but in college for shorter works i always thought handwriting, typing, and reading it aloud was my allowing my brain to hit a problem in three different (not more, different) ways before / even if ever being in front of another human being (or now their non-union AI equivalent)
i still feel writing something down helps with memory formation, in part because i remember the act of writing it down. cocktail napkins and whiteboards way better than ipads also
i remain obsessed with Pilot v5 rolling ball extra fine; i would say Pilot g2s are for proles but the outcome if you leave it in the dryer is far less severe
How about you stop taking things so personal instead and let others steer more.
reply