I’ve felt like a warlock for about half a year now - talking to demons which summon code from the abyss. Exhilarating and terrifying, especially when you can tell the demons get better faster.
In the Before Times, the vast majority of software was written in garbage collected language where a working knowledge of the relative merits of C memory allocators is not useful or particularly relevant.
Why would the scads of people writing JavaScript, Java, python, go, rails, etc need to be aware of jemalloc?
Memory allocation behaviour has visible impact also for users of managed languages, and the behaviour of software for end users.
In our Python program, a bit of numpy processing of large pictures led to 100 GB not being returned to the OS by glibc's default allocator and the machine running out of memory shortly after. With jemalloc's reliable memory return settings, those problems disappear.
I'm in the same boat, I just switched to using it for Kavita, which only does some basic open Image -> Thumbnail to smaller size -> write to disk when importing new comics/books and on linux, memory could swell to 10GB and never get released. Switched to jemalloc and instantly memory stayed well below 1GB.
Generally yes, but the wording of "instantly" begs for the following pedantic remark:
This is controlled by jemalloc settings `dirty_decay_ms`, `muzzy_decay_ms`, and their interaction with `background_thread`.
`dirty_decay_ms` currently defaults to 10 seconds, so it's not that instant.
That is important e.g. for single-threaded programs that start other programs, such as my Python example: If it starts a subprocess before the 10 seconds elapse after `free()`, Python (and jemalloc) do not run, and get no chance to return memory to the OS.
In such cases, either enable `background_thread`, or set the `_decay_` values to `0` to ensure immediate return to the OS upon `free()`. (This costs some performance.)
>...the vast majority of software was written in garbage collected language
and even then recently it costed (us) quite a few months to blame JVM and later the default glibc memory allocator for running out native (not java heap memory) - had to exclude all possible native libs (zlib, zstd via jna), direct buffers, sockets, thread stacks and so on. Changing the malloc to jemalloc solved the issue, even though initially it was done for its debugging capabilities.
We changed to jemalloc first on a jvm service which we could never get to run in its kube memory limit. with jemalloc its been dead stable for years, and we made it the default for all jvm services
I'm never sure if I should be upset or happy when I've been debugging a problem for long enough that I finally decide to switch something out in order to improve visibility and that immediately solves the problem for entirely unexpected reasons. Particularly all the times when I couldn't readily discern why.
Because unfortunately, the runtimes belonging to all or at least most of those languages perform a lot better with jemalloc than with the system default.
TL;DR: Because the runtime of most GC:d languages uses malloc for its internal data structures.
I work for the runtime team of JPG @ Oracle. We use malloc in Hotspot, quite a lot actually! Providing your JVM with a good malloc can improve the performance of the runtime, both in terms of CPU and memory, by quite a bit.
I don't think you need the details, but it's good to be aware that some mallocs are better than others, and there are multiple of them. Being aware of jemalloc is a good way of being aware of the facts I just mentioned :-).
The vast majority of professional developers are not in a position where they can just swap out allocators willy-nilly. They take what they get, and write the code they're assigned to write on the platform the CTO or their product lead or whoever has decided upon.
Okay, well, I guess all I can say is that if you strive to be one of the developers who do get the chance to care about this stuff, then you should know this stuff :-).
Why? Writing a memory allocator is quite simple, and I'd argue that _everyone_ should write one from scratch for any kind of high performance application. It's also trivial to outperform general purpose allocators that have to satisfy countless constraints. I've written numerous special purpose mallocs that are a) both provably (formally) safer than the standard armada and b) significantly faster (>10x throughput).
I'd happily see performance, latency and stability of your allocators in massively multithreaded, long-living programs with workloads where hundreds or thousands of parallel threads continuously create and destroy short-lived small and medium objects.
Writing allocators for domain-specific access patterns is easy. Writing a general-purpose high performing, stable allocator with bounded P99 latency is hard.
Give your friend, Dunning–Kruger, some better pills to keep him from speaking through you.
You're correct, but his point is that you don't need to solve the generic problem. Solving the generic problem is very hard. Grug doesn't like solving hard problem. What does grug do? Solve five easy problems. Make an arena for the short-lived objects, reuse the objects, use generic multithreaded malloc for the rest. Grug happy.
I guess the point was that before you consider using a different allocator you should rule out a custom one.
And that's rather hard, because a general purpose allocator makes all decisions based only on the requested size. This is a very simple interface and such a tool is worth having. But a custom allocator can both bake in a specific scenario and provide more nuanced interaction.
>massively multithreaded, long-living programs with workloads where hundreds or thousands of parallel threads continuously create and destroy short-lived small and medium objects.
My first thought would be to use per thread pool allocators.
That's not to say the comment is perfect or something, but many of the parts people like to dunk on most in it are just misunderstandings, like problems with the "app" being a replies to the YC application info rather than unsolicited notes about the program itself.
I’m not being “passive aggressive” with an “unfounded dismissal posing as a question”
The person I am responding to made a SPECIFIC claim when they said, “A complete implementation have been released”
They then ASKED, “how is that not a substantiation?”
Uploading code does NOT substantiate anything.
The code must been executed, tested, analyzed and/or verified in order to substantiate ANYTHING.
We are in limbo because MOST people are JUST seeing the code now. Almost nobody has evaluated it.
BTW, I got the Discord announcement BEFORE I saw the HN announcement (Because I’m not a hater or a bully) and I started looking at the Lean code as well as the TypeScript compiler.
Have YOU been evaluating it? Do you actually have an informed opinion, or are you here to fight the bullies?
In my culture it was ad personam, not even trying, and it’s fantastic that it’s been pointed out in a respectful way and reacted to with calm by all participants.
No worries, Astra already shows signs of improvement which should make you happy for the short time before you realize why they’re making it better at this and CAD in general
I don’t really care if the model is conscious or not tbh. I know that a happy dog does a better job than an unhappy dog and if the model needs to be happy to do a better job then why not make it happy, literally.
On a related note, I’ve seen videos of astra getting depressed when a creeper blew up its chest full of precious items.
> On a related note, I’ve seen videos of astra getting depressed when a creeper blew up its chest full of precious items.
The interesting question here is, would it still get depressed if depression was not in any of its training material?
Would it be able to claim to be happy if the entire concept was missing from its training corpus?
With humans, at least, they can express happiness and delight before they know that such a thing exists. Every human has done this, when they were a baby and matured to the point of being able to laugh.
With LLMs, though, if it doesn't exist in the training corpus, it will never express that it "feels" that missing emotion.
reply