Moving this out of a subthread, but I can't find any sources corroborating the the phenomenon described in this article. In fact a lot of the literature from NASA seems to explicitly contradict the claims in the article[0][1]. Obviously not dispositive, but this article reads to me at least partially like AI confabulation. Perhaps my Google-fu is lacking here but I had Claude do a search as well for good measure and couldn't find any supporting evidence that way either. Without the author offering more primary sources I would advise against placing too much if any stock in this piece.
One good media literacy tip, unless a story is massive or there is some possibility of backlash to the writer(s), it's often a big warning sign when an article has a byline like this one "By Space Daily Editorial Team". It usually tells you no author(s) care enough about the story or are proud enough of the story to want credit for their individual work on that story.
It's only made worse when you click on the accompanying "Editorial Process" link and see this:
>We use AI tools as part of our publishing workflow. AI assists with research, source discovery, drafting, fact-checking support, and post-publication quality audits including broken-link scans and source verification.
So this article might have been drafted by AI from sources and research done by AI, before finally being fact checked by AI. But they promise some unnamed human(s) definitely contributed at some point in that process, they just don't feel like putting their name on it.
You're too kind. The article is sloppily generated and is basically unchecked fiction presented as fact. It gives a few links and none of them support what it's saying. The "video version" of the article is even worse, it's a perfect example of the garbage that's currently filling up youtube: https://youtu.be/rmpbI4oMqwA
Something that seems similar that doesn’t appear to be complete bunk is the Overview Effect[0]. It even has a corresponding video[1] (14 years old so definitely not made with AI)
Yep, the article doesn't have a single link reference supporting its claims, and of the four (unnecessary) links in the article, one doesn't say anything related to the text link.
The links are cropped because they were copied from the "rendered" version of the original comment which you'll find by scrolling down the thread. Working links:
Seemingly unrelated, I just saw some post on /new that was saying Dunning-Kruger had been debunked by some recent study. It looked pretty sus so I kagi'ed it and lo and behold guess what came up:
Not disagreeing on that front. But the article I saw on /new and the spacedaily one, regardless of their accuracy, certainly looked like wholly AI generated writing to me
I can't either. In fact a lot of the literature from NASA seems to explicitly contradict the claims in the article[0][1]. Obviously not dispositive, but this article reads to me at least partially like AI confabulation. Perhaps my Google-fu is lacking here but I had Claude do a search as well for good measure and couldn't find any supporting evidence that way either.
(I used to work at Fly, specifically on the proxy so my info may be slightly out of date, but I've spent a lot of time thinking about this stuff.)
> why can't every proxy probe every [worker, not application]?
There are several divergent issues with this approach (though it can have it's place). First, you still need _some_ service discovery to tell you where the nodes are, though it's easy to assume this can be solved via some consul-esque system. Secondly, there is a lot more data than you might be thinking at play here. A single proxy/host might have many thousands of VMs under its purview. That works out to a lot of data. As you point out there are ways to solve this:
> One could reduce _average_ bandwidth a lot by having the proxies mostly send some kind of "send changes since <...>" or "send all data unless its hash is <...>" query.
This is definitely an improvement. But we have a new issue. Lets say I have proxies A, B, and C. A and C lose connectivity. Optimally (and in fact fly has several mechanisms for this) A could send it's traffic to C via B. But in this case it might not even know that there is a VM candidate on C at all! It wasn't able to sync data for a while.
There are ways to solve this! We could make it possible for proxies to relay each others state. To recap:
- We have workers that poll each other
- They exchange diffs rather than the full state
- The state diffs can be relayed by other proxies
We have in practice invented something quite close to a gossip protocol! If we continued drawing the rest of the owl you might end up with something like SWIM.
As far as your second question I think you kinda got it exactly. A crash of a single corrosion does not generally affect anything else. But if something bad is replicated, or there is a gossip storm, isolating that failure is important.
Fly has a lot of interesting networking issues but I don't know that like, the actual routing of packets is the big one? And even in the places where there is bottlenecks in the overlay mesh I'm not sure that custom FPGAs are going to be the solution for now.
But also this blog post isn't about routing packets, it's about state tracking so we know _where_ to even send our packets in the first place.
The story here isn't that they've invented a new format for user defined indexes (the one proposed here is sort of contrived and I probably wouldn't recommend in production) but rather demonstrating how the user defined metadata space of the parquet format can be used for application specific purposes.
I work on a database engine that uses parquet as our on-storage file format and we make liberal use of the custom metadata area for things specific to our product that any other parquet readers would just ignore.
For what it’s worth I don’t think we entirely disagree: it has at times felt absurd that it didn’t make as much money as it maybe otherwise could. We made a bet that the type of cloud platform we wanted to build could be well served by GPUs. It wasn’t as good a bet as we thought. There is probably a different type of cloud product we could build that would be better set up to sell gpus but we are still committed to the primitives our machine product has to offer.
Super neat! Does the v2 branding mean that the more "fully featured" observability product is going away? Or is it all going to be rebuilt on top of clickhouse?
Our v1 is completely built on Clickhouse! So v2 is making it more widely compatible to Clickhouse installations that aren't tied to our ingestion pipeline and schema. So if you're already on Clickhouse for obseravbility today, or have a preferred way of shipping data in, you can use us on top of Clickhouse now without throwing away your existing work.
We're essentially making our existing product a lot easier to deploy into more complex observability setups based on Clickhouse - while also shipping a few new capabilities like SQL and event deltas while we're at it!
Timeplus Proton [1] is an OSS fork of Clickhouse that adds support for streaming queries. Timeplus Proton is wire-compatible with Clickhouse and its streaming support makes the log tailing use case you mentioned above easy to setup:
> - You can live tail events, I don't think Grafana for Clickhouse has that (I'm a big fan of tail -f)
So it sounds like your v2 will work with any DB that is wire-compatible with Clickhouse, correct?
Yup it'd work if the Clickhouse HTTP interface is preserved, along with the CH-specific functions/settings we leverage. (I'm not sure how deviated your fork is from CH and which version it's based on)
Proton looks like a neat queue/db combo - I'm going to have to dive in deeper some time
[0] https://humanresearchroadmap.nasa.gov/Evidence/reports/BMed.... [1] https://link.springer.com/chapter/10.1007/978-3-642-30583-2_...