I respect what you are saying, and agree our LLM are just superficial pseudo communication engines. Theories of intelligence differ, but I don't think human intelligence, i.e. IQ of 100 is really all that far ahead of LLM. Just talk to your average person. We are easily wrong. We have bias and error. We repeat and amplify untruths and make mistakes.
It could be fewer breakthoughs than you think to get there. Not because the tech is so advanced, but because we are not that special.
Super-intelligence is just degrees better.
I had a lot of fun talking to one about how we could fully electrify a hobby farm without batteries, by using a high voltage DC overhead wire on a cable pulley system. Everything from electrical engineering, to cable selection, sourcing vendors and costs.
Mistakes were made, but it was genuinely helpful at times. I was impressed at how far it has come.