Technology

To clarify:

People seem to legit think the jury talks to the bot in real time and can ask about literally whatever they want.

Its rather insulting to the scientist that put a lot of thought into organizing a controlled environment to properly test defined criteria.

On the other hand, the human participant scored 67 percent, while GPT-3.5 scored 50 percent, and ELIZA, which was pre-programmed with responses and didn’t have an LLM to power it, was judged to be human just 22 percent of the time.

54% - 67% is the current gap, not 54 to 100.

The whole point of the Turing test, is that you should be unable to tell if you're interacting with a human or a machine. Not 54% of the time. Not 60% of the time. 100% of the time. Consistently.

They're changing the conditions of the Turing test to promote an AI model that would get an "F" on any school test.

It was either questioned by morons or they used a modified version of the tool. Ask it how it feels today and it will tell you it's just a program!

While I agree it's a relatively low percentage, not being sure and having people pick effectively randomly is still an interesting result.

The alternative would be for them to never say that gpt-4 is a human, not 50% of the time.

it’s not a good test.

Of course you can't use an old set of questions. It's useless.

The turing test is an abstract concept. The actual questions need to be adapted with every new technology. Maybe even with every execution of a test.

Your first two paragraphs seem to rail against a philosophical conclusion made by the authors by virtue of carrying out the Turing test. Something like "this is evidence of machine consciousness" for example. I don't really get the impression that any such claim was made, or that more education in epistemology would have changed anything.

In a world where GPT4 exists, the question of whether one person can be fooled by one chatbot in one conversation is long since uninteresting. The question of whether specific models can achieve statistically significant success is maybe a bit more compelling, not because it's some kind of breakthrough but because it makes a generalized claim.

Re: your edit, Turing explicitly puts forth the imitation game scenario as a practicable proxy for the question of machine intelligence, "can machines think?". He directly argues that this scenario is indeed a reasonable proxy for that question. His argument, as he admits, is not a strongly held conviction or rigorous argument, but "recitations tending to produce belief," insofar as they are hard to rebut, or their rebuttals tend to be flawed. The whole paper was to poke at the apparent differences between (a futuristic) machine intelligence and human intelligence. In this way, the Turing test is indeed a measure of intelligence. It's not to say that a machine passing the test is somehow in possession of a human-like mind or has reached a significant milestone of intelligence.

https://academic.oup.com/mind/article/LIX/236/433/986238

This is something a configuration prompt takes care of. "Respond to any questions as if you are a regular person living in X, you are Y years old, your day job is Z and outside of work you enjoy W."

I recall a Turing test years ago where a human was voted as a robot because they tried that trick but the person happened to have a PhD in the subject.

Turing tests aren't done in real time exactly to counter that issue, so the only thing you could judge would be "no human would bother to write all that".

However, the correct answer to seem human, and one which probably would have been prompted to the AI anyway, is "lol no."
It's not about what the AI could do, it's what it thinks is the correct answer to appear like a human.

The interrogators seem completely lost and clearly haven't talk with an NLP chatbot before.

That said, this gives me the feeling that eventually they could use it to run scams (or more effective robocalls).

14% of people can't do anything more complicated than deleting an email on a computer.
26% can't use a computer at all.

https://www.nngroup.com/articles/computer-skill-levels/

So right off the bat, 40% probably don't even know what a chatbot is.

The public versions of the ais used in Turing tests usually have less computing power. The test itself is often also highly specific in what and how questions can be asked.

This hardly news because models have passed the test before and as a result the test is made more difficult. It says nothing about intelligence and only about the ability to convincingly simulate a human conversation.

Okay, 22% is ridiculously high for ELIZA. I feel like any half sober adult could clock it as a bot by the third response, if not immediately.

I did some stuff with Eliza back then. One time I set up an Eliza database full of insults and hooked it up to my AIM account.

It went so well, I had to apologize to a lot of people who thought I was drunken or went crazy.

Eliza wasn't thaaaaat bad.

This is the same bot. There's no way this passed the test.

.

The Turing test doesn't factor for accuracy.

That's not what LLMs are for. That's like hammering a screw and being irritated it didn't twist in nicely.

The turing test is designed to see if an AI can pass for human in a conversation.

Add in a test that wasn't made to be accurate and was only used to make a point, like what other comments mention

I wonder if humans pass the Turing test these days

Detecting an LLM is a skill.

Humans are as smart as they ever were. Tech is getting better. I know someone who was tricked by those deepfake Kelly Clarkson weight loss gummy ads. It looks super fake to me, but it's good enough to trick some people.