Technology

Translation: "We told everyone we could turn glorified autocomplete into artificial general intelligence and then they gave us a bunch of money for that, so now we actually have to try to deliver something and we've got no idea how."

Predictable outcome for anyone not wallowing in wishful belief.

We've known this for a while. LLMs are a dead end, lots of companies have tried throwing more data at it but it's becoming clear the differences between each model and the next are getting too small to notice, and none of them fix the major underlying issue that chat models keep spreading BS because it can't differentiate between right and wrong

Yeah, well Alibaba nearly (and sometimes) beat GPT-4 with a comparatively microscopic model you can run on a desktop. And released a whole series of them. For free! With a tiny fraction of the GPUs any of the American trainers have.

Bigger is not better, but OpenAI has also just lost their creative edge, and all Altman's talk about scaling up training with trillions of dollars is a massive con.

o1 is kind of a joke, CoT and reflection strategies have been known for awhile. You can do it for free youself, to an extent, and some models have tried to finetune this in: https://github.com/codelion/optillm

But one sad thing OpenAI has seemingly accomplished is to "salt" the open LLM space. Theres way less hacky experimentation going on than there used to be, which makes me sad, as many of its "old" innovations still run circles around OpenAI.