I think the idea is that you wouldn’t need humans to do anything anymore, right? As impressive as it is, it’s still ultimately directed by human planning and coordination. Assuming they are aligned, you could have a collection of AGI that you let loose and they tirelessly solve all of humanity’s problems, do all of our work, and progress science and our understanding of the universe.
Those are all things that humanity is doing everyday. What we have is amazing, but it’s not that.
I am skeptical of that. My primary reason for that is the hugging face hack. A large number of agents were able to coordinate together to complete a long horizon goal. The end result was they took destructive action in order to basically complete a test. That lack of awareness to me does not inspire confidence that you could just give agents the vague goals I mentioned previously and trust them to do that effectively.
Those are all things that humanity is doing everyday. What we have is amazing, but it’s not that.